From the context of government e-services & application projects across around the world, the ability of an ICT department to create and deliver quality assurance (QA) services on time & budgets to its respective stakeholders has a significant impact on improved customer service and delivering measurable business value.
Successful government CIO’s & QA Departments deliver and release services which have gone through rigorous scrutiny & inspections from design, function, technological & user-centric perspectives.
To clear & add up to quality assurance perspective, it does not mean testing. Rather, it is described as a function with the responsibility to ensure that these services & applications will meet its intended objectives & requirements – functional, date, budget, etc. Testing is important, but it is different, it is rather quality control which also forms an important component of the quality assurance process.
Quality Assurance Department’s job is to be a customer before a customer.
Step 1: Properly Defined Objectives & What Success looks like?
The primary objective of a QA department is to ensure successful project execution. While this may seem quite obvious, many enterprises struggle to follow through on it. Many QA departments are established without a well-defined objective or with an objective such as:
- Improving quality.
- Achieving CMM level X.
- Implementing a new methodology.
- Process improvement and so on….
These are good secondary objectives, but they are not the same as ensuring successful projects.
There are many situations where overemphasis on a single laudable goal is counterproductive. Focus too much on any one goal (e.g., date, budget, user requirements, etc.) and it is highly probable that meeting only a specific goal is at the expense of the success of the project.
For example, many government enterprises have caused themselves significant problems by rushing to release new e-services & applications to its stake- holders before they were ready. This can be confirmed by the United Nations E-Government Services Survey, conducted around the world, which issues challenges and obstacles faced by even the most successful ICT departments in different geographies.
Defining success is not always easy, It can also be very situational. However, successful e-services & application release is defined by some of the entities in the following manner:
Government of Singapore: 30% Increase in the adoption of the e-services & applications utility by the intended user (Citizens, Businesses, Residents & Visitors)
Government of United Kingdom: Customer satisfaction increase by 15% and reduction in their usage of other channels (Such as Kiosks, Call Center, Government Office Visits etc.) due to the release of new or existing e-services & applications.
Government of South Korea: Utility of the e-services & application by the user to reduce their time to complete a particular activity by 3 Days.
Government of Sweden: Usage of the e-services & application by the user to reduce the costs of completing a certain government related activity.
The trade-off between cost, target dates, quality, and user satisfaction can differ significantly for every government entity, and even for every project. Success may be very target date driven on one project, quality driven on another project, cost, or user satisfaction.
Success is usually some combination of these factors and other factors. Achieving success may require different strategies depending on a number of factors including: the size and experience of the project team, the nature of the requirements, the technology, user involvement, etc. In some cases success may mean minimizing financial loss – e.g. killing a project before too much money has been spent.
Step 2: Defining QA ‘s Responsibilities & Reporting Structure.
Ensuring successful projects releases based on Step 1, requires a QA department to work closely and in synergy with development teams & project managers. This view of a QA department’s responsibilities goes beyond defining the responsibilities of the QA group to be just defining process and conducting reviews.
Ensuring project success requires QA staff to work more closely (and perhaps earlier in the project) with project management than do most QA departments.
Success means that QA must take responsibility (with the project manager) for the success of the project.
QA Department’s Major Responsibilities:
- A Quality assurance department provides an import- ant check and balance on the process, a QA department is bestowed by the responsibility to identify the problem early. It is not uncommon for more urgent responsibilities to distract management from paying as close attention to some projects as it would In these situations, a QA department should provide an important service by monitoring projects.
- A QA department’s responsibility is to provide an in- dependent, objective & unbiased
- A QA department is the appropriate central that needs to be responsible for
Details on some of their Day-to-Day Activities & Responsibilities
- Obtain user stories, requirements, functional design, internal design specifications, or other available / necessary information.
- Obtain budget and schedule requirements.
- Determine project-related personnel and their responsibilities, reporting requirements, required standards and processes (such as release processes, change processes, etc.).
- Determine project context, relative to the existing quality culture of the product / organization / busi- ness, and how it might impact testing scope, approaches, and methods.
- Identify the application’s higher-risk and more important aspects, set priorities, and determine scope and limitations of tests.
- Determine test approaches and methods – unit, integration, functional, system, security, load, usability tests, whichever are in scope.
- Determine test environment requirements (hard- ware, software, configuration, versions, communications, etc.)
- Determine test-ware requirements (automation tools, coverage analyzers, test tracking, problem / bug tracking, etc.)
- Determine test input data requirements.
- Identify tasks, those responsible for tasks, and personnel requirements.
- Set initial schedule estimates, timelines, milestones where feasible.
- Determine, where appropriate, input equivalence classes, boundary value analyses, error classes.
- Prepare test plan document(s) and have needed reviews / approvals.
- Write test cases or test scenarios as needed.
- Have needed reviews / inspections / approvals of test cases / scenarios / approaches.
- Prepare test environment and test-ware, obtain needed user manuals/reference documents / configuration guides/installation guides, set up test tracking processes, set up logging and archiving processes, set up or obtain test input data.
- Obtain/install/configure software releases
- Perform tests.
- Evaluate and report results.
- Track problems/bugs and fixes.
- Retest as needed.
- Maintain and update test plans, test cases, test environment, and test-ware through life cycle.
“QA is management’s responsibility” It is highly important for the senior management to understand their role in the QA process, then a QA department is likely to succeed. In the life of every QA department, bureaucracy is a common affair, and QA’s reporting structure can either paralyze or empower them to deliver and create immense value for the organization.
Successful ICT departments have their QA departments empowered by giving them access & reporting structure directly to the following roles based on an organizations unique culture:
- Chief Information Officer (CIO).
- Head of Applications & e-Services Delivery.
- Chief Technology Officer (CTO).
Step 3: Hiring the Right Competency & Team.
The hiring and staffing implications for an ICT QA department & team can make or break the success parameters of a successful QA program. Some of most important criteria should be carefully looked into while hiring, staffing and creating a competent team:
Experience is highly important, the designed structured QA should have relevant experience either working in and around a government ICT department. The team should be able to understand the is- sues that are most important to project success (e.g., user involvement, budget, calendar, technology, etc.) and have the ability to define clear processes that will result in success of the department.
The Quality Assurance department must be staffed by people who will be respected by the project managers, development teams and senior management for the skills, capabilities, emotional quotient and mind-set they bring in the organization. They should have the ability to connect, establish relationships and communicate value so they can develop an effective working relationship with all other relevant stakeholders.
Required Competencies in a QA Department.
- Requirement Analysis & document verification
- Front-end testing
- Backend / data validation testing
- Sufficient time available to devote to testing
- Ability to identify all possible positive and negative scenarios
- Familiar with testing techniques
- Ability to perform multiple cycles of regression testing
- Ability to understand the business Proficiency
- Ability to test 24 hours a day
- Ability to automate tests and reduce cycle time
The above mentioned competencies are highly linked to the demand section of the overall IT Strategy, which will dictate the number of projects passing every quarter through the QA department.
The number of personnel required in the QA department will be based on the varying needs of the government ICT operations and the e-Services & applications released based on a robust plan.
Step 4: Setting metrics for QA department’s Accountability.
Responsibility Comes With Great Accountability
If the QA department is responsible for ensuring success, it should also be held accountable. If senior management does not hold the QA department ac- countable for unsuccessful projects, it is an indication that management does not believe in the value of a QA department or the capability of the QA staff.
The real challenge for CIO’s & IT Leaders in government ICT departments is to know, how can both development project managers and QA staff be held accountable for the success of a project without any conflicts of interest. The answer is a concept called “joint responsibility for success” In practical terms what this means that If a project was unsuccessful, the project manager and QA Staff would be called into a meeting with the CIO & IT Leadership on a frequent basis and discuss challenges which need to be overcome jointly.
Most Government ICT organizations and projects have measures that are important to them – usually cost and target date. If managed just by cost and tar- get date, the measurement is quite weak. Encouraging teams to meet cost and target date objectives at the expense of user satisfaction, system reliability, or some other important objective is a commonly seen mistake.
An integrated set of measures that:
Are aligned with the project’s and the organization’s objectives.
Provide a balanced perspectives.
Provide insight & accountability to the appropriate person.
Are natural byproducts of the process.
Properly done, measurement can make life easier for the QA department, development teams, project managers, IT leadership and senior management.
Measurement helps drive the process improvement priorities. It also makes the QA department’s job reviewing projects much easier. When starting off with a new QA department, initially the challenge will be that the metrics are not being computed, or if they are, project managers may not be anxious to share them. However, If the metrics are indeed important to senior management, the QA department will have the clout necessary to get project managers to compute and report the metrics.
Step 5: Setting Standards & Processes that are followed and are sufficient yet flexible.
One of the most important roles of a QA department is establishing a consistent process for the organiza- tion and working with project teams to adapt the pro- cess to their unique circumstances.
This role puts the QA department in a position to transfer best practices developed by one project to other projects. It will also identify process gaps. In mature organizations, following the right process be- comes a habit and is part of the organization’s culture. In less mature organizations, there are several situations a strong QA department can address:
Scenario 1: Projects are not consistently using the standard process where it would be appropriate and effective.
Due to new complexities with new products, services and technologies on board, it is likely that many, if not most, Government ICT projects will not follow the existing standard process, despite it being appropriate and effective. This is one area where the new QA department can bring in quick improvements specially in cases where there is a relatively immature development group. In many such scenarios it has been observed that QA department can add great value very quickly.
Scenario 2: The standard process is not appropriate for a particular project.
No matter how good the process is, there will be situation where it does not seem to fit a current project. If project managers & development teams decide for themselves not to follow the standard process (i.e., the process is a set of guidelines not standards), then the QA department might lose control. In such scenarios Development teams & project managers usu- ally tend to take short cuts and the standard process will not be consistently followed when they are appropriate and effective.
If the QA department imposes that the process always be used (i.e., the process is a set of inflexible standards, not guidelines), then the QA department may cause a project to be unsuccessful. Best practice to follow on this problem is to have the QA department and the project manager decide at the beginning of a project what deviations from the standard process are appropriate. Project management can deviate from the standard process, but they need the QA department’s buy in first.
Scenario 3: New process is needed to meet a project’s objectives.
No matter how good current methodologies, process and standards, they will not be sufficient for your most demanding projects. Additional and/or different process/approaches are frequently necessary for very large, high-risk projects or projects using new technology. These projects should drive enhancements to the existing process. A QA department is in a good position to identify the need for new process and ensure the process is enhanced to meet the needs of the projects.
Documentation in Process Requirements Even in mature organizations, the need to improve the process is ongoing. Fortunately, today’s technology makes it much easier to develop and disseminate a new process. In the past, the common approach was to document a process in a binder and distribute the binder to the staff. (Actually, some methodologies had many binders.) Producing, updating, and distributing the process was very expensive. Today, the preferred approach is to make the process accessible on a corporate intranet. While the costs to define the process may not have changed dramatically, the costs to maintain, distribute, and access the process have declined dramatically.
Imperfect or flawed processes cause most system problems. Thus, to prevent these problems, an organization must improve its process. If a QA department is to be held accountable for project success, it must have the ability to create, enforce, and improve the process.
This role puts the Quality Assurance department in a position to transfer best practices developed by one project to other projects.
Step 6: Setting Up QA Department’s Infrastructure.
Now as seen in the above steps, most of the Government ICT departments create a basic inventory of Infrastructure that is required to support for the QA department to achieve its objectives. This includes, technological infrastructure, documentation templates and QA tools which fit the orga- nizational culture and needs.
- Test Plan
- Hardware & Software Documentation
- Test Case Repository (spreadsheets or any other tools)
- Load/Performance Tools
- Issue Tracking Plan & Tools
- Templates for Test Plan, Test Case, QA Sign-Off
- QA Metrics Templates
- Communication & Escalation Plans
Measurement helps drive the process improvement priorities. It also makes the QA department’s job reviewing projects much easier.
Step 7: Setting up Communication Plan & effectiveness.
Risk is inherent in software development & E-Services releases. The risk of delaying a system and forgoing possibly significant benefits. Since risk cannot be eliminated, the goal is to understand risk so that prudent decisions can be made early in the project by the right group to minimize risks.
QA is really a form of risk management. There are many ways a QA department can contribute to a better understanding and management of the risks facing a project. Integrating risk management into the software development process and using measurement to communicate the risks to the project team and to management are two areas where a QA department can add significant value. Since many soft- ware development risks are commonly encountered, techniques that identify them and address them can easily be built into the process.
Best Practice Examples:
- The risk associated with a cost or target date estimate should be included in the estimate documentation along with a clear communication of the source of the risk (Such as vague requirements, scarce re- sources, ) to senior IT Leadership & management. Assumptions to be included with an estimate is an- other way to better communicate the risk associated with an estimate early on in the project.
- The risk associated with requirements should be understood. If the requirements are not well under- stood, more user involvement will be warranted. If scope creep is a risk, a robust change control process will be needed and this needs to be communicated effectively.
Metrics can be another very powerful way to identify, communicate, and address risks. One very common situation occurs when a system is nearing its planned release date – especially if a third party is developing the system. Developers are under pressure to get the system installed and will focus on the number of open defects. While this is a useful measure, it does not provide a balanced perspective of the risk involved with releasing the application or services.
For example, the defect arrival rate (how many defects have been found during a specified period of time) can provide a better indication of whether or not the system is ready to be released, If the developers found and closed 100 defects during the last week, the open defect count might be zero. However, the defect arrival rate says that the risk of releasing is still unacceptably high.
Step 8: Sustained Improvement with the right Training & Development.
The IT Leadership & Senior Management needs to put up a robust plan to educate and improve the learning of the new QA departments. This department will stumble in the beginning and have its own set of challenges which are very unique to its respective Government ICT culture and environment. However setting up frequent training inside and outside the organization can keep the team’s spirit high and motivated to perform their role more effectively.
Some Government ICT organizations also are on a look out for new technologies and products that can help the team automate certain processes to reduce their time in successful completion of QA department’s tasks.
The idea of having these new tools in the organization also increases the understanding of what other successful entities are doing but at the same time empowers QA department to deliver faster and better services to the organization.
An art honed by the the senior management to be a customer, think like a customer and act like a customer can open insights which can lead to creating & running a successful Quality Assurance Department.