Quality Assurance's Role in Software Acquisition

Purchased software presents a serious challenge to Quality Assurance's ability to test. Typically the cost of the acquired software is considerable, but can represent a dramatic savings over developing it in-house. That is if it works and contains all necessary functionality. The purchased software may be a custom application or a commercial off the shelf (COTS) package. Regardless, Quality Assurance should participate in the acquisition process to increase the likelihood that the outcome will be acceptable.

There is a scene that most purchasers of software are all too familiar. The vendor asks for a meeting. During the meeting he holds up a piece of paper from the requirements document and says, "What did you mean by this?" Regardless of your answer, you will next hear, "That's not how I interpreted it. This is going to take longer and cost more than we originally discussed." This usually occurs so late in the development effort that we are unable to do much more than acquiesce. In essence we renegotiate the contract whenever it is to the vendor's advantage. We are so late into the process there is no other option.

We often attribute the increased costs and time to an incomplete specification document, a poorly worded contract, mismanagement, bad luck, or any number of reasons that may or may not be correct. But these situations occur so frequently, we must start to address the solutions and not resign ourselves to doing business this way.

Most software contracts emphasize payment schedules and deadlines without actually stating unambiguously when the job will be done. These problems are also the opportunities for Quality Assurance.

Here are some step-by-step recommendations that may help to resolve these problems and document the contribution of Quality Assurance.

Step 1: Early QA Involvement

QA should be brought into the acquisition process as early as possible. They should assist the Business Analysts in identifying the business requirements of the application. QA may start the construction of a traceability matrix by building a preliminary set of functional test scripts to correspond to each requirement.

Step 2: The Request for Information (RFI)

The RFI is distributed to a large number of potential vendors. It identifies the primary system objectives. Vendors are asked "Can you give us what we want?" or "Do you have something that can be simply customized for us?" All vendors want to be in the running and respond positively to the RFI. They usually indicate that they can already handle about 90% of the requirements. While the high percentage sounds very good, it really tells us very little.

To get a better response and to give the vendors a level field, QA should include a preliminary test plan with the RFI. It should contain at least one test to cover each major piece of functionality in the desired system. The potential vendors are then asked to indicate which tests they can pass with their current version of the system.

Step 3: The Request for Proposals (RFP)

The vendor field is reduced to those companies that indicate that they can pass the tests found in the RFI. The limited field is now given the RFP with a test plan that is fleshed out much more than the preliminary plan found in the RFI. The RFP is now asking for proposals for a system that will pass all of the tests in the test plan.

Step 4: The Contract

In addition to the normal clauses in the contract, there should be a test plan. QA should have the system test plan completed and it should be an integral part of the contract. The vendor is being asked to produce a system that will be considered "done" when it passes all of the tests in the test plan.

Now when you are asked what a particular statement means, you simply give an explanation and indicate which test or tests in the plan will provide the necessary clarification. This process should go a long way towards avoiding the renegotiation of the contract.

Issue 1: Payment Schedule

The contract will probably indicate a schedule of payments that reflects the progress of the job. While we certainly do not want to pay when there is no progress, always remember that our objective is to pay. If the vendor does not perform, we will fail to meet our business objectives and that may be far more costly than the software. Find ways to help the vendor be successful.

Issue 2: Testing

QA will test the product when it is delivered and everyone will expect the test to be successful. QA can ensure that there will be a high probability of success if they start testing with the vendor as early as possible. This means that potentially a QA Analyst may work at the vendor site.

Issue 3: Automated Testing

If your organization will be responsible for maintaining the new application, discuss testing with the vendor. Try to include their test plans as one of the deliverables. If the vendor will be automating the testing process, you probably want them to include the scripts with the other deliverables so they can be the foundation for your regression test. You will have to purchase the appropriate automated test tool to use their scripts.