Reviewing Contracts: Acceptance Tests
By Susan Ronning, PE
One of the most important contractual items of a new or replacement radio communications system is the acceptance test. Acceptance tests come in different names, styles and formats, but in the end, the final acceptance is meant to achieve one goal—validation that the radio system works where, when and how it was designed. There’s no one test that determines final acceptance. It’s the aggregation of all the tests done at various stages throughout the system’s implementation.
The naming conventions used to identify the various stages of acceptance tests differ by vendor, consultant and agency. To make matters worse, each radio communications system is unique; therefore, the associated tests must be specific to the system under test. However, even with these differences, all projects can follow the same stages; therefore, the various acceptance tests should also follow these project stages.
Clarity is vital to achieving project success. As early as possible, the customer—or consultant on the customer’s behalf—and vendor should clearly share their understanding of the project stages and their associated tests and deliverables. Time and cost are always factors. Tests require time commitments from both the vendor and the customer.
The following tests are listed in the order they would normally occur. Included is the number of times that test would be expected to be performed:
1. Factory acceptance test—once per system component;
2. Staging acceptance test—once per system;
3. Site acceptance test—once per physical location;
4. Coverage verification test—once per system, region or subsystem;
5. Customer acceptance test—once per system; however, may be divided into logical regions in a phased system deployment; and
6. Final acceptance test—once per system or phased system deployment.
Each testing stage requires a set of test procedures, defined expectation of results, required data captured and an analysis of data. Satisfactory completion and customer acceptance of all the above stages of testing constitutes final acceptance. Each test stage is further described below.
Stage 1: factory acceptance test
The factory acceptance test, sometimes known as box testing, is performed on each system component (i.e., base stations, controllers, routers, subscriber radios, etc.) at the manufacturer location once the equipment has been built. Performance specification sheets should ship with the equipment or be delivered to the project manager for incorporation into final system documentation.
Stage 2: staging acceptance test
The staging acceptance test may also be known as system staging. This test is performed in a lab or warehouse environment on a full or a partial representative configuration of the system. The purpose of this test is to demonstrate system features, functions and failure scenarios in a controlled environment. The staging acceptance test provides the opportunity to test equipment and subsystems that are integrated into the radio communications systems. These subsystems may include dispatch consoles, special system features and third-party equipment.
Standards compliance and/or interoperability tests with third-party equipment, such as consoles, logging recorders and subscriber units, should also be performed at this stage.
Although network interfaces may be tested in this environment, actual throughput and bandwidth constraints often cannot be tested; however, they should be simulated if possible.
Stage 3: site acceptance test
The site acceptance test may also be known as power-up or cold-start tests. This test is performed in the field, on a site-by-site basis, once all equipment designated for a particular location is installed and designated complete. The purpose of the site acceptance test is to validate that all equipment at an individual site is installed properly and operates as expected.
Site equipment may include, but is not limited to, radio base stations, radio controller equipment, RF antennas, transmission lines, ground lines, lightning arrestors, backhaul interfaces, network terminals and/or console dispatch positions. Site locations are type-designated based on their primary function—RF repeater site, radio controller site, dispatch center, etc.
The site acceptance tests are developed specific to the site type. Each site type requires a subset of tests based on associated subsystems, which may include primary and initial power system, mechanical system, site access and monitoring system, RF antenna system, radio equipment, network and backhaul and others. The test procedures, expectation of results, data captured and data analysis are all specific to their subsystem.
Stage 4: coverage verification test
The coverage verification test, also known as coverage acceptance test or simply coverage testing, is performed when the sites have completed their site acceptance testing and, therefore, have been fully installed with final RF systems. The purpose is to capture the actual RF transmission performance and analyze it against the predicated area performance and coverage guarantees of the system design as stated in the negotiated contract.
Any changes to an RF site, such as to types or locations of antennas, may have a direct impact on the performance of the radio system as a whole. The propagation characteristics of a site are affected by the antenna make/model, azimuth and physical tower placement. The propagation characteristics of a radio system are based on each site’s individual contribution to the system. Any changes that affect the system design can, in turn, affect the system performance and coverage. These coverage changes may or may not affect the coverage guaranty, depending on the structure and wording of the contract. All parties need to be aware of the requested or required change and its effect on the system coverage and performance guarantee.
The coverage verification test is developed specific to the radio system modulation. The recorded data may be delivered audio quality (DAQ), received signal strength (RSSI), bit error rate (BER) or a combination as appropriate.
Stage 5: customer acceptance test
The customer acceptance test may also be known as a field acceptance test or functional testing. This test is performed in the field once all associated sites are installed (site acceptance tests complete), and backhaul connectivity is available to all associated system locations and stable. The purpose of this test is to demonstrate system features, functions and failure scenarios in the system in its final configuration.
Dispatch consoles and subscriber radios must be configured with their final templates and programming in order to perform this test. Additionally, subsystems that have been integrated into the radio communications systems must also be tested. In particular, network performance, including bandwidth availability and microwave fades, affects the radio communications system. Network performance should be verified, tested and recorded prior to starting a system acceptance test in order to ensure the radio system network requirements have been met and any problems arising during the test are not due to backhaul or bandwidth issues.
Stage 6: final acceptance test
The final acceptance test may also be known as final sign-off or 30-day test or burn-in period. It kicks off a period of time designated for the customer to use the system and identify issues. Issues are identified as either major or minor and are recorded in a final acceptance punch list. Major punch list items may cause a restart of the test time period, depending on the contract provisions.
Once the final acceptance test completes its time frame with no major punch list items identified, both the vendor and customer have completed all stages of testing of the newly installed radio system. The customer signs for final acceptance. The radio system may then move into the warranty phase.
About the Author
Susan Ronning, PE, has experience as a radio system vendor, consultant and customer and has been involved in all aspects and phases of a project, from inception to final acceptance. She holds a BSEE and MBA and is an active member of IEEE, APCO International and Civil Air Patrol. Ronning is a senior systems engineer for Tait Radio Communications.
This article originally appeared in September 2012 Public Safety Communications.