|
|
|
|
Your browser does not support CSS. Please upgrade to the current version
of Internet Explorer or Netscape Navigator.
|
Testing, Quality Assurance, and Security Techniques
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| I. Introduction: Defect Detection or Defect Prevention? | ||||||
| The objectives of testing are examined and the responsibilities of testers at all levels are defined. Quality concepts are introduced along with a list of success factors that will facilitate decision-making at each stage of the testing process. Numerous tests may be conducted during the SDLC (System Development Life Cycle) and each is the primary responsibility of a specific group. As each group completes a testing phase a formal transition process is initiated before the next group begins. This section identifies entrance and exit criteria by phase for each component of the development team. The object is to eliminate as much testing overlap as possible. | ||||||
| (a) Objectives | ||||||
| (b) Observations | ||||||
| (c) Impediments to Quality Exercise | ||||||
| (d) Role of the Tester | ||||||
| (e) Responsibilities | ||||||
| (f) Early Testing vs. Late Testing | ||||||
| (g) Quality Assurance Assessment | ||||||
| (h) Quality Issues and Elements | ||||||
| Quality Improvement Suggestions | ||||||
| Quality Tools and Steps | ||||||
| (i) Opportunities to Improve the Testing Process | ||||||
| (j) System Development Life Cycles | ||||||
| Waterfall SDLC | ||||||
| Spiral SDLC | ||||||
| (k) Phase Objectives | ||||||
| (l) Measuring Performance | ||||||
| (m) Reliability Metrics | ||||||
| (n) Testing Success Factors | ||||||
| (o) Product Development and Testing Phases | ||||||
| II. Major Software Development and Testing Issues | ||||||
| Successful software projects have clear specifications highlighting system objectives, and serve as the vehicle for communicating all information to the individuals participating in the development process. Modern development practices recognize that changes to the plan may be required during the System Development Life Cycle and seek to manage the changes. This section emphasizes the contribution of testers during the SDLC. | ||||||
| (a) Preparing Specifications | ||||||
| Writing User Manuals | ||||||
| Writing Skills | ||||||
| (b) QA / QC Responsibilities | ||||||
| (c) Reviewing Project Specifications | ||||||
| Exercise: Sample Project Specification | ||||||
| Specification Review Example | ||||||
| Specification Problems? | ||||||
| Specification Problem Classification | ||||||
| Contents of a Specification | ||||||
| More Specification Guidelines | ||||||
| Tables in Specifications | ||||||
| (d) Scripts and Cases | ||||||
| (e) Unit vs. System (or Acceptance) Testing | ||||||
| (f) Confidence in Testing | ||||||
| (g) Positive and Negative Testing | ||||||
| (h) Blind Testing | ||||||
| (i) Unit-level Test Scripts | ||||||
| (j) System-level Test Scripts | ||||||
| (k) Managing Change | ||||||
| (l) Levels of Testing | ||||||
| (m) Responsibilities by Test Type | ||||||
| III. Test Methodologies and Checklists | ||||||
| Testing methodologies enable testers to compute their test coverage and have confidence that all requirements will be tested. Both black and white box techniques are used to demonstrate methodologies. These testing approaches are demonstrated and reinforced with exercises. The use of methodologies in testing is an essential element of a quality assurance organization. | ||||||
| (a) Setting Test Objectives and Identifying Tests | ||||||
| (b) Test Planning | ||||||
| (c) Methodologies | ||||||
| Test Coverage Computation | ||||||
| Black Box Testing (Closed Box) | ||||||
| White Box Testing (Open Box) | ||||||
| (d) Boundary Value Analysis | ||||||
| Specification Example with Boundaries | ||||||
| Exercise: What Are the Boundary Conditions? | ||||||
| (e) Path Analysis (Cyclomatic Complexity) | ||||||
| Path Analysis Example | ||||||
| Creating a Flow Chart or Data Flow | ||||||
| Data Flow Example | ||||||
| Decision Tree | ||||||
| Complexity Analysis with Java Code | ||||||
| Path Analysis Example (Closed Box) | ||||||
| Test Scripts | ||||||
| Exercise: Data Flow (Accounting). | ||||||
| Exercise: Data Flow (Airline). | ||||||
| Exercise: Data Flow (Vesting). | ||||||
| (f) Decision Tables | ||||||
| Binary Arithmetic | ||||||
| Decision Table Example | ||||||
| Function Table | ||||||
| Exercise: Decision Table (Telephone). | ||||||
| (g) State Machines | ||||||
| (h) State Transition | ||||||
| (i) Factor Analysis | ||||||
| (j) OATS — Orthagonal Array Testing Strategy | ||||||
| OATS Example | ||||||
| Using OATS | ||||||
| (k) Pairs and Magic Squares | ||||||
| Odd Order Templates | ||||||
| (l) Embedded Systems | ||||||
| (m) Additional Tests | ||||||
| Employee Last Name Checklist | ||||||
| Table and Array Testing | ||||||
| Date Edit Checklist | ||||||
| Screen, Button, and Character Entry Checklists | ||||||
| IV. Test Planning | ||||||
| Testing begins with a plan that unambiguously states the objectives. A suitable methodology is selected to provide adequate test coverage and to deliver the desired level of confidence that the software will perform as advertised. Testing is treated as a dynamic process that may continue after delivery and will certainly play a role in future system modifications. Appropriate record keeping is initiated and maintained through the life of the product. | ||||||
| (a) Unit Testing (Early Testing) | ||||||
| White Box Test Case Sources | ||||||
| Sample Unit Test Plan Table of Contents | ||||||
| Unit Testing Scenario | ||||||
| (b) Integration Testing and System Testing | ||||||
| (c) System / Acceptance Testing | ||||||
| Sample System (or Acceptance) Plan Table of Contents | ||||||
| Sample System (or Acceptance) Test Script | ||||||
| (d) Possible Test Plan Elements | ||||||
| Sample System / Acceptance Test Plan | ||||||
| (e) Creating the Regression Test | ||||||
| Regression Test Alternatives | ||||||
| Traceability Matrix | ||||||
| (f) Usability Testing | ||||||
| (g) Palm Compliance Testing Checklist | ||||||
| (h) Stopping Rules for Testing | ||||||
| (i) Estimating with Function Points | ||||||
| (j) How Do I Estimate the Testing Effort? | ||||||
| (k) Data Dictionaries | ||||||
| (l) Approaches to Testing | ||||||
| Top-Down | ||||||
| Bottom-Up | ||||||
| (m) Regression Testing | ||||||
| (n) Alternatives to Testing | ||||||
| (o) Test Notebook | ||||||
| V. Defect Prevention | ||||||
| If we know what has gone wrong in the past we should be able to avoid repeating the problems in the future. This section examines some of the past lessons, and suggests approaches to resolving problems. It recognizes that the chief responsibility of testing is defect prevention — not defect detection. | ||||||
| (a) Checklists | ||||||
| (b) Functional Specification Defects | ||||||
| (c) Design Defects | ||||||
| (d) Coding Defects | ||||||
| (e) Testing Defects | ||||||
| (f) Coding / Testing Rules | ||||||
| VI. Components of Web Applications | ||||||
| We discuss the components that make up a typical web application. The various components need to be tested individually and also in their entirety. Identifying and documenting the components will assist in the testing process. Developers need to build the components for maintainability and testability, test the components, and document their interactions. The multi-tier aspect of web applications force testers to change some of the fundamental assumptions that both developers and testers make about applications and their interfaces. | ||||||
| (a) Physical Parts — Architecture and Technology | ||||||
| (b) Logical Parts — The Business Side of Computing | ||||||
| (c) Various Standards and the Effect on Web Applications | ||||||
| (d) Mapping Physical-to-Logical Parts | ||||||
| VII. Overview of Web Application Testing Issues | ||||||
| An accurate determination of the issues involved in testing will guide the testing process. This determination should include the strategies for our development and testing process. When the web testing process is new to an organization or a player, the web testing issues must be forefront in the thought process. These issues and checklists will become the framework of the instruction for the rest of the workshop. | ||||||
| Exercise: Diagram a simple web application. | ||||||
| (a) Checklist: Included Items | ||||||
| Computer Hardware Descriptions | ||||||
| Computer Software Descriptions | ||||||
| Connection Determinations | ||||||
| Configuration Issues | ||||||
| Servers | ||||||
| Clients | ||||||
| Operating Systems | ||||||
| Locations | ||||||
| Onsite | ||||||
| Power Issues | ||||||
| Cabinet Monitoring | ||||||
| Fault Tolerant — RAID Arrays | ||||||
| Drive Reliability Testing | ||||||
| Physical Security | ||||||
| Offsite | ||||||
| Service Level Agreements (SLAs) | ||||||
| Security | ||||||
| Equipment | ||||||
| Disaster Recovery | ||||||
| (b) Checklist: Critical Issues When Testing for Web Applications | ||||||
| (c) Checklist: Common Items for All Test Plans | ||||||
| (d) Checklist: Critical Issues in Any Test Design | ||||||
| Discussion Items: | ||||||
| (a) Designing Test Cases That Fit with Web Applications | ||||||
| (b) Different Test Types and the Associated Merits, Pitfalls and Costs | ||||||
| VIII. Important Quality Metrics for Web Applications | ||||||
| Some attributes are common to all web applications; however, others only apply to a select number of web sites. You will learn some common measurements that are used to assess quality in a web application. You will practice techniques for developing appropriate metrics for your own applications. | ||||||
| (a) Checklist: Common Metrics Used for Evaluating Web Applications | ||||||
| (b) Portability | ||||||
| (c) Security | ||||||
| Monitoring | ||||||
| Updates/Patches | ||||||
| User Activities — Passwords | ||||||
| (d) Keeping an Eye on the Future | ||||||
| Survivable | ||||||
| Durable | ||||||
| Modifiable | ||||||
| Maintainable | ||||||
| (e) Responsiveness | ||||||
| User Needs | ||||||
| Maintenance | ||||||
| Upgrades | ||||||
| Testing Issues | ||||||
| Security Breaches | ||||||
| Program Flow | ||||||
| Hardware Issues | ||||||
| (f) Usability | ||||||
| (g) Scalability | ||||||
| (h) Identifying the Best Quality Metric for the Organization | ||||||
| Exercise: Creating corporate metrics. | ||||||
| IX. Testing the User Interface of a Web Application | ||||||
| Every web designer, indeed everyone who uses the web, knows the importance of a good user interface. The interface often is a matter of personal preference, but the needs of the many must outweigh the desires of a few. Testing the interface needs to be as objective as possible, and sometimes that is extremely difficult. The interface gives the corporation one opportunity to satisfy the need or the user will disconnect quickly. You will learn how to evaluate and test a user interface. | ||||||
| (a) Checklist: Quality Issues for the User Interface | ||||||
| (b) Issues | ||||||
| Portability | ||||||
| Browser Compatibility | ||||||
| Usability | ||||||
| International | ||||||
| Wireless Devices and User Interface | ||||||
| (c) Verify | ||||||
| (d) HTML Syntax | ||||||
| (e) Links | ||||||
| Update Links | ||||||
| Internal Links | ||||||
| External Links | ||||||
| Graphics | ||||||
| Icons | ||||||
| (f) Web Content | ||||||
| Accuracy | ||||||
| Consistency | ||||||
| Delivery | ||||||
| Effectiveness | ||||||
| Intuitive | ||||||
| Usefulness | ||||||
| User-Friendliness | ||||||
| (g) The Effect of User Interface Design on Test Case Design | ||||||
| Exercise: Designing the user interface test plan. | ||||||
| (h) Tools for User Interface Testing | ||||||
| (i) Testing User Documentation and Training Material | ||||||
| Strategy | ||||||
| Readability | ||||||
| Size | ||||||
| Printed vs. Online | ||||||
| X. Testing the Database of a Web Application | ||||||
| The focal point of the web application resides in the database. The database must be able to organize and deliver information to the clients who are the requesters. Testers must prove that the database is correctly organized, that data collected by the user interface is properly stored, and that queries generated by the user interface produce accurate and timely responses. Data integrity issues and back-end reporting issues are also discussed. | ||||||
| (a) Standard Testing Issues | ||||||
| Adding/Saving | ||||||
| Modifications/Updating | ||||||
| Deleting | ||||||
| Audit Trails | ||||||
| Backups/Restores | ||||||
| (b) Checklist: Quality Issues for the Database of a Web Application | ||||||
| (c) White Box Database Testing | ||||||
| (d) Black Box Database Testing | ||||||
| (e) Types of Back-Ends — Are There Differences? | ||||||
| Characteristics of DB Servers | ||||||
| Non-Proprietary | ||||||
| dBASE | ||||||
| MS Access | ||||||
| ASCII | ||||||
| Proprietary | ||||||
| Sybase | ||||||
| MS SQL Server | ||||||
| My SQL | ||||||
| Aestiva | ||||||
| ORACLE | ||||||
| Co-Existence of Various DBs | ||||||
| Interoperability | ||||||
| (f) SQL Query Tests | ||||||
| Stored Procedures | ||||||
| Information Exchanges | ||||||
| Parameter Storage | ||||||
| (g) Referential Integrity | ||||||
| (h) Database-Driven | ||||||
| Assessing the Current Needs | ||||||
| Planning for the Future | ||||||
| Security Features | ||||||
| The Database | ||||||
| The Information | ||||||
| The Error Recovery Process | ||||||
| XI. Testing the Business Logic of a Web Application | ||||||
| Testers must prove that a web application provides the intended business services. The web application is designed and deployed to resolve business issues. The flashiest web application is useless if it does not fulfill the business objective. The tester must be able to identify the business function, proving that the web application workflow is correct and that error conditions are identified and properly handled. | ||||||
| (a) Business Objective | ||||||
| Critical Success Factors | ||||||
| Deployment Factors | ||||||
| Identification of User Groups | ||||||
| Evaluation Criteria for Successes | ||||||
| (b) White Box Tests | ||||||
| Functionality | ||||||
| Business Needs | ||||||
| (c) Black Box Tests | ||||||
| Environmental | ||||||
| Use Cases | ||||||
| (d) Positive Tests | ||||||
| Path Analysis | ||||||
| Risk Assessment | ||||||
| (e) Negative Tests | ||||||
| Boundary Analysis | ||||||
| Error Handling | ||||||
| Error Recovery | ||||||
| (f) Assessing the Number of Test Cases Needed | ||||||
| Exercise: Calculate the number of tests required for one component of a web application. | ||||||
| XII. Integrating Web Applications | ||||||
| Large web applications consist of many components. The order in which these components are added to the web application can affect the amount of work the test organization must perform. Many web applications will have to interact with the existing IT infrastructure. This process will involve intensive preparation and consistent involvement and interactions. The difficulty involved is increased if the application has external interfaces, or the hosting is with an external source. | ||||||
| (a) Scheduling the Integration | ||||||
| Development | ||||||
| Testing | ||||||
| Documentation | ||||||
| User Acceptance | ||||||
| Delivery to the Client | ||||||
| Moving from Conventional Sources to the Web | ||||||
| Outside Agencies | ||||||
| Offsite Hosting Services | ||||||
| Installation of Software/Hardware | ||||||
| External Interfacing | ||||||
| (b) Integration Success Factors | ||||||
| Cycles | ||||||
| Components | ||||||
| Business Functions | ||||||
| Logical Groupings | ||||||
| (c) Integration Strategies and Their Effect on Test Load | ||||||
| (d) Integration with Legacy Systems | ||||||
| XIII. System Testing and the Web Application | ||||||
| The loads a web application must deal with are extremely volatile and notoriously difficult to predict. The assessment and monitoring of these conditions will be beneficial or detrimental to a web application. The tester needs to ensure that the application will meet the current need and that there are monitoring techniques included to handle future requests. | ||||||
| (a) Checklist: Defect Points of Web Applications | ||||||
| Definitions | ||||||
| Performance | ||||||
| Load | ||||||
| Stress | ||||||
| Parallel | ||||||
| Appropriate Timing | ||||||
| Page Timeouts | ||||||
| Maintaining Sessions | ||||||
| Responsibilities | ||||||
| Critical Components | ||||||
| Bottlenecks | ||||||
| Availability | ||||||
| Tools | ||||||
| (b) Workload Considerations | ||||||
| Demands — Establishing Current Peaks/Valleys | ||||||
| Demands — Assessing Future Resources | ||||||
| Concurrent Activities | ||||||
| (c) Network Issues | ||||||
| (d) Browser Issues | ||||||
| (e) Identifying Transactions | ||||||
| Completed Transactions | ||||||
| Interrupted Transactions | ||||||
| Recovery Operations | ||||||
| Reentrant Transactions | ||||||
| Non-Reentrant Transactions | ||||||
| (f) Testing the Install/Uninstall Process | ||||||
| Flash | ||||||
| Real | ||||||
| Shockwave | ||||||
| (g) Handling the Failures | ||||||
| Server Mirroring | ||||||
| Fail Over Hosting | ||||||
| Alternate Hosts | ||||||
| (h) Scalability Issues | ||||||
| Round Robin DNS | ||||||
| Load Balancing | ||||||
| XIV. Security Issues for Web Applications | ||||||
| Understand the importance of building security into your web applications. Examine software requirements, design, and code as early as possible in an application to expose security vulnerabilities. Add appropriate cases to your test designs to explore software with a new awareness of security issues. | ||||||
| (a) What is Security Testing? | ||||||
| (b) Risk Based Security Testing | ||||||
| (c) Test Strategy and Planning | ||||||
| (d) Security Metrics | ||||||
| (e) Checklist: Common Security Issues | ||||||
| (f) Security Violations | ||||||
| (g) Penetration Testing | ||||||
| (h) Performing Security Tests | ||||||
| (i) Analysis of White Box and Black Box Testing | ||||||
| (j) Backup and Recovery Issues | ||||||
| (k) Security Tools | ||||||
| (l) Responsibilities for Security | ||||||
| Business Analysts | ||||||
| Programmers | ||||||
| Testers | ||||||
| Managers | ||||||
| Database Administrators | ||||||
| XV. Secret Stuff | ||||||
| Building the application with testable metrics will enable the business analyst to make better business web decisions, testers to make better test management decisions, and the programmers to provide better web applications. The web application should be self-monitoring with the ability to do defect tracking within the application itself. The testers should be evaluating these opportunities within the context of testing the application. | ||||||
| (a) Outsourcing | ||||||
| Issues | ||||||
| Programming | ||||||
| Testing | ||||||
| Equipment | ||||||
| Vendor Lock | ||||||
| (b) Continual Management Process | ||||||
| Audit Trails | ||||||
| Self-Monitoring Techniques — Auto-Defect Recording | ||||||
| Stress | ||||||
| Peak Load Monitoring | ||||||
| Performance (Ping, Tracert) | ||||||
| Remote Monitoring Services | ||||||
| (c) Auto-Backup Generation | ||||||
| (d) Options for Client Defect Reporting | ||||||
| (e) Assessment of Risk Factors | ||||||
| Visibility | ||||||
| Cost | ||||||
| Continuity of Operations | ||||||
| New Code vs. Old Code | ||||||
| Complexity | ||||||
| Programmers | ||||||
| Analysis of the Paths | ||||||
| (f) Dealing with the Scope Creep Issues | ||||||
| CSF — Critical Web Application Success Factors | ||||||
| Phasing the Web Application Delivery Schedule | ||||||
| Customer Patterns | ||||||
| Development Patterns | ||||||
| (g) Guaranteed Failures | ||||||
| Documentation | ||||||
| Error Detection and Recovery Procedures | ||||||
| Consistency Across the Application | ||||||
| Web Automation Plan | ||||||
| Over-Reliance on the Tools | ||||||
| Written Application Test Plans | ||||||
| Unknown Application Coverage | ||||||
| Inability to Perform System Tests | ||||||
| (h) Suggested Readings | ||||||
| XVI. Software Tools for Testing | ||||||
| Some testing is impractical or impossible without automation. While automated testing is clearly a direction that most organizations should pursue, it is necessary to examine the potential benefits and the problems associated with automating tests. | ||||||
| (a) Automated Testing Considerations | ||||||
| (b) Test Tools | ||||||
| XVII. Getting Started | ||||||
| Theory, concepts and experiences do not always point us in the correct direction for future application testing efforts. All organizations must develop a strategy to start the process and then refine their techniques. The Software Engineering Institute's Capability Maturity Model offers significant guidance for developing and testing systems. | ||||||
| (a) Capability Maturity Model | ||||||
| (b) CMM Level 1 — Initial | ||||||
| (c) CMM Level 2 — Repeatable | ||||||
| (d) CMM Level 3 — Defined | ||||||
| (e) CMM Level 4 — Managed | ||||||
| (f) CMM Level 5 — Optimized | ||||||
| XVIII. Glossary | ||||||
Welcome
About Us
On-Site Workshops Request Information