With the advent of consumer applications, it was quickly recognized that traditional development practices would not work. The linear stages of the Waterfall model could not keep up with the changing user needs. By the time that software had fallen over the waterfall, it was either far removed from what the user actually wanted or another company had got there first.
That?s assuming that the user needs had been accurately captured in the first place, which was not common. Users frequently had to settle for second-best software that did not accurately reflect their needs. Today?s experience-driven consumers, by contrast, will simply look for a better alternative offered by another vendor.
Redefining the User Experience
Interest in Agile methodologies has continued to rise in part due to the ability to engage the user earlier and more frequently while becoming more reactive to change. Any misunderstanding of the users? desired functionality could then be identified earlier as developers work iteratively to deliver software that reflects accurately the changing user needs.
In addition to more regular opportunities for user feedback, the apps themselves are producing more data than ever before, which can be analyzed to identify opportunities for improvement in the software. Meanwhile, DevOps is bringing tools across the entire development lifecycle into closer alignment so that these large amounts can more easily be collated and analyzed.
As a result, analytics has become an increasingly popular and effective aid to decision-making in software development. This topic is tackled in the latest of a series of articles written by testing guru Paul Gerrard.
Testing and Analytics
As Paul notes, analytics does not affect the fundamental goal of testing, which still gathers and analyzes information to inform decision-making. A difference in the discipline that Paul calls test analytics, however, is that it analyzes the entire lifecycle and extends beyond pre-production testing.
Test analytics places an emphasis on application monitoring where the data produced by pre-production testing is analyzed as well as the data produced within production and operations. The results are then fed back into support testing?s core product, or the information passed on to stakeholders.
?Shift Left, Shift Right?
To extend beyond pre-production testing, testing needs to shift right as well as left, leveraging the insights of production and operations and reflecting these back in test cases and the system requirements.
One way to do this is to leverage the large amount of data that users input into a live system as well as the large amount of data that comes out. This data can be analyzed, perhaps using association rule-learning, reverse-engineering how users are using a system in practice.
One possible application of these insights in testing is knowledge elicitation. The insights from operations might reveal previously unknown or unknowable aspects of a systems logic that can then be reflected in more accurate, comprehensive tests. If model-based testing is used, these insights can be reflected iteratively in the model, generating an updated set of tests that more closely reflect the system under.
Another application is in risk-based testing. Live systems data can be analyzed to identify the relative likelihood that certain areas of a systems logic will be exercised during operations. Tests can then be generated to satisfy empirically defined risk thresholds, maximizing the likelihood that defects will be detected and eradicated before the user finds them. This is especially valuable when testing is pressed for time and resources.
Analytics Informs Decisions and Validates Them
Analytics can be used to support more informed decision making about the direction of software. It can also be used to iteratively define tests that more accurately reflect the desired user experience. Application monitoring and analytics are, therefore, invaluable tools for both decision-making and for assuring that those decisions have been successfully implemented in development. They can enable organizations to stay ahead of the game, delivering applications that reflect constantly changing user needs.
