Numbers and graphs are ok but...
As testers and test managers, we are often working in or in close contact with the development organisation and colleagues who all speak the same language. We monitor the progress using the metrics agreed upon, such as test progress, covered and residual risk, number of defects, their severity and so on. While this reporting makes perfect sense to most of us, the business may not feel properly informed to take important decisions in regards to a Go or No Go to production.
This uncertainty will often lead to lengthy meetings, where we will be adressing the different aspects of the report to different stakeholders. Multiple stakeholders also means multiple agendas and levels of technical insight, så again we need to translate our reports to suit the needs of the business. I like the face to face meetings as much as the next one, but often time is a luxury we cannot afford.
This communication gap requires 2 versions of the test report, each targeted at different groups of stakeholders. A manual “translation” would require additional time which is a sparse commodity. I have also often wondered if one version aimed at the business would suffice, but decision makers in DevOps need more detailed information, so this idea was scrapped.
Translating the reports
The idea came to me in the weirdest of ways, and thinking of it brings a smile to my face. I was reading a LinkedIn post about the increased number of AI created materiel and how writing is becoming impersonal and soulless. This post inspired me to do exactly what the article warned about, using AI to write. My take on it however, was a bit different as my idea was to use AI to transform my test reports to something with value for the business.
To test my hypothesis of my “Report Translator”, I needed a test report. Since I did not have a real report and would never post it online either, I asked ChatGPT to create a test report based on a template found online. I also asked for sample data, so I had something to work with. Once i had the report, I wrote a simple prompt, asking ChatGPT for a business centric version of the report.
Our department manager does not have the technical insight
to decipher the test report and has asked you to create a business centric version
of this report. please do so
Below are the unedited versions of the original report and the translated version. In a real situation, the reports would have a different format and the content of the reports would be determined by the data reported on.
Basic test report
Test Cycle: System Test
EXECUTED
PASSED: 110
FAILED: 20
(Total) TESTS EXECUTED: 130
PENDING
IN PROGRESS: 5
BLOCKED: 3
(Sub-Total) TEST PLANNED: 138
| Functions | Description | % TCs Executed | % TCs Passed | TCs Pending | Priority | Remarks | Defects Identified |
|---|---|---|---|---|---|---|---|
| New Customer | Check new Customer is created | 100% | 90% | 0 | High | 2 Failures | 2 |
| Edit Customer | Check Customer can be edited | 100% | 95% | 0 | High | 1 Failure | 1 |
| New Account | Check New account is added | 100% | 85% | 0 | High | 3 Failures | 3 |
| Edit Account | Check Account is edit | 100% | 80% | 0 | High | 4 Failures | 4 |
| Delete Account | Verify Account is deleted | 100% | 100% | 0 | High | No Issues | 0 |
| Delete Customer | Verify Customer is Deleted | 100% | 90% | 0 | High | 2 Failures | 2 |
| Mini Statement | Verify Mini statement is generated | 100% | 100% | 0 | High | No Issues | 0 |
| Customized Statement | Check Customized Statement is generated | 100% | 85% | 0 | High | 3 Failures | 3 |
Summary:
- Total Test Cases Executed: 130
- Total Passed: 110
- Total Failed: 20
- Total Defects Identified: 15
- Pending Test Cases: 5
- Blocked Test Cases: 3
Key Observations:
- Most failures were encountered in Edit Account, New Account, and Customized Statement features.
- Defect severity analysis:
- 5 Critical defects (impacting core functionalities)
- 6 Major defects (causing functionality issues but with workarounds)
- 4 Minor defects (UI/validation issues)
- Blocked test cases were mainly due to environment issues and dependencies on other teams.
- Urgent resolution required for Edit Account and New Account functionalities due to their high failure rates.
Next Steps:
- Immediate defect triage for high-priority failures.
- Re-execute failed test cases after fixes are applied.
- Resolve blocked issues to proceed with pending test cases.
- Prepare for next test cycle with additional regression coverage.
Prepared by: Obi-Jan
Date: March 21st 2025
Business faced
Business-Centric Weekly Test Report
Overview:
The testing cycle for the system test phase has been completed, covering critical business functionalities such as customer management, account management, and financial transactions. A total of 130 test cases were executed, with 110 passing and 20 failing, resulting in an overall test success rate of 85%. There are 5 test cases still in progress and 3 blocked due to external dependencies.
Key Business Insights:
Overall System Stability:
- The system is functioning as expected in most key areas, with an 85% success rate.
- Critical functions such as customer creation, account deletion, and mini-statement generation are working well.
Issues Requiring Immediate Attention:
- The Edit Account and New Account features experienced the highest number of failures, impacting user operations for modifying and adding new accounts.
- Failures in the Customized Statement generation may affect customer communications.
- A total of 15 defects were identified, with 5 categorized as critical (impacting business continuity).
Current Risks to Business Operations:
- Delayed account modifications may lead to customer dissatisfaction and operational inefficiencies.
- Blocked test cases may indicate potential challenges that could delay the final deployment.
Next Steps:
- The development team has been notified, and defect fixes are being prioritized based on business impact.
- Once fixes are applied, failed test cases will be re-tested to confirm resolution.
- Blocked cases will be addressed by collaborating with relevant teams to remove dependencies.
Conclusion: While most functionalities are performing well, there are areas that require urgent fixes before the system can be deemed ready for production. The testing team will provide an update next week on the progress of defect resolutions.
Prepared by: Obi-Jan
Date: March 21st 2025
Conclusion so far...
I am fully aware that this probably not what your reports would look like, but that was also not the “scope” of this test. I wanted to test if a simple prompt on an uploaded report could produce anything remotely useable.
As i mention in my post AI in software testing – Analysing a user story, hallucinations and bias has an impact on the result you get when working with LLMs. In this example I wanted a fast feedback, so I just accepted the output as is. There are several ways to improve the result, either by working with special instructions in your LLM or by creating a dedicated GPT for the purpose. I will be covering this in future post, so stay tuned.
Happy Testing