APPROACHES OF IMPLEMENTATION CHATGPT IN A SOFTWARE TESTING AUTOMATION PROCESS

APPROACHES OF IMPLEMENTATION CHATGPT IN A SOFTWARE TESTING AUTOMATION PROCESS

Myrzakulova Altynshash

 Student, Kazakh-British Technical University,

Kazakhstan, Almaty

ABSTRACT

Automation of software testing is a crucial aspect of the software development process. It helps to improve the accuracy, speed, and efficiency of testing, and reduce the cost and effort associated with manual testing. With the advent of artificial intelligence (AI) and recently released Chat Generative Pre-trained Transformer (ChatGPT), new opportunities for automating software testing have emerged. The use of AI in software testing has the potential to revolutionize the quality assurance process, enabling organizations to deliver high-quality software to their customers. Along with implementing ChatGPT into an ad hoc testing process we can implement the following into CI/CD. This article aims to explore the potential of ChatGPT in software testing and examine how it can be applied in a software development process - both manual and automation - along with describing benefits of using ChatGPT3 in software quality assurance in order to encourage its adoption in the software development community.

Keywords: ChatGPT, software testing, quality assurance, Artificial Intelligence, software testing automatization.

Introduction

Artificial Intelligence (AI) has the potential to revolutionize the software quality assurance process. AI-based systems can automate various tasks, reduce manual effort, and improve accuracy, speed, and efficiency in software testing. AI algorithms can be trained to perform automated testing, report bugs and other issues, create test cases along with other test artifacts and can also be integrated into the software development pipeline to automate tasks such as code quality checks, testing, and deployment. By utilizing AI in software quality assurance, organizations can achieve faster and more effective testing, improve software reliability and usability, and reduce the cost and effort associated with manual testing. One of the recent innovations in an artificial intelligence field is a Generative Pre-trained Transformer (ChatGPT-3) [1] provided by OpenAI.

ChatGPT, as a product of advanced artificial intelligence and language processing, can be used in the software testing process in several ways. In 2019  Gao, Tao, Jie and Lu held a two year experiment, based on their Software Quality Assurance (SQA) class where students have been involved in testing projects for various mobile apps that incorporate machine learning. They have employed traditional testing methods and tools and during the testing process, they encountered numerous questions, not only for manual testing, but also for AI based [2]. Some of the questions, arised in that research, could be solved by ChatGPT. To begin with, it can be handy while generating new or optimizing already existing test cases. Moreover, in perspective it could provide generated test data and even write automated test cases applying a hybrid approach which is formed from combined use of ChatGPT with more traditional testing methods and techniques, such, for example - manual testing, can help to maximize the benefits of ChatGPT while also mitigating its potential limitations. Training how to use ChatGPT more efficiently can help to maximize the impact of the technology while also minimizing the need for additional staffing.

This paper will focus mainly on approaches of how ChatGPT-3 functionality could be applied in everyday SQA pipeline in both manual and automation routine. Section 2 would discuss advantages and disadvantages of AI generated test cases. Section 3 and Section 4 would observe methodology, where former would focus on implementation manual test cases to a native mobile app based on production design and latter would be about implementation from the automation perspective.

Pros and Cons of AI Generated Test Cases

Attempts to implement AI generated test data have already been done [3]. At the very beginning, it was presented in a form of UML-diagram [4]. Later, more advanced approaches have been developed, for example, Yalla and Sunil [5] have created a sentence generator powered by Natural Language Processing (NLP), which can be utilized as a test input for conversational AI bots. This can be done through the use of automated testing tools or frameworks like Selenium, which are popular in the testing industry. Dipti Belsare and Manasi Bhate [6] provided a literature review for the same NLP topic.

Implementing AI generated test cases in a real life project can have both advantages and disadvantages. On the positive side, using a language model like ChatGPT can significantly increase the speed and efficiency of creating test cases, as it eliminates the need for manual effort in writing and verifying test scenarios. Furthermore, it can provide a large number of test cases in a short amount of time, thus improving test coverage and reducing the likelihood of software bugs going unnoticed. On the other hand, there is a risk that the generated test cases may not cover all possible edge cases, or may contain inaccuracies or errors. In addition, the interpretability and maintenance of test cases generated by a machine learning model can be challenging, as it can be difficult to understand the reasoning behind the generated test scenarios. Therefore, it is important to carefully evaluate the trade-offs and to continuously monitor the results of using ChatGPT generated test cases in a work project.

Additionally, it is important to consider the expertise and resources required to implement and use ChatGPT effectively. Proper training of the model and ensuring its compatibility with the software being tested can be time-consuming and may require specialized skills. Furthermore, the quality of the generated test cases depends heavily on the quality and quantity of data used to train the model, and may require regular maintenance and updating. Despite these challenges, the benefits of using ChatGPT generated test cases can be substantial, particularly in terms of increasing efficiency and improving the overall quality of the testing process. In general, the decision to implement ChatGPT generated test cases in a work project should be based on a careful consideration of both the potential benefits and the costs, and a thorough evaluation of the project’s specific needs and resources.

Another consideration is the potential ethical implications of using ChatGPT generated test cases. As the model is being trained on large amounts of data, there is a risk that it may pick up biases or inaccuracies present in the training data, thus training data should be of high quality and free of inaccuracies. This can result in generated test cases that are unfair, discriminatory, or otherwise unethical. Therefore, it is important to carefully evaluate the data used to train the model and to monitor its outputs regularly to ensure that they align with ethical and cultural norms.

Methods and Approaches

For a manual QA test cases engage a major role and make a deep integration into the Agile process, therefore they couldn't be neglected and in a long term run could save a bunch of time, not only in development, but also as onboarding for newcomers. Test case actualization and prioritization usually took at least one day for QA (depends on project and sprint). According to Agile everyone writes tests [7]. Test cases itself are commonly implemented in a requirements analysis step before development and along with or after a design review and eventually used also as a test documentation. Main approach here is to test how convenient and reliable ChatGPT3 would be in this step.

Creating a test case with ChatGPT

Since now ChatGPT3 is unable to process images, one of the possible solutions is to simply describe the design on your own. On Figure 1 could be observed an example user interface design of a car profile in a users garage. The aim here is to imitate user behavior as he wants to enter a service history.

Design of the whole user flow won't be provided, but the entering point will be to open the app, preconditions: user should have at least one car in the garage and should be authorized. Then we “feed” these instructions to ChatGPT requesting to write a test case according to our instructions. So the request was: write a test case to enter the service history of the auto preconditions - user should be authorized and has at least one car in the garage. Received output was the follows:

Figure 1. «Example of a UI design»

Test Case: Entering Service History for a Car

Preconditions:

• The user must be authorized and have an active account.
• The user must have at least one car added to their garage.

Test Steps:

1. Log in to the system using a valid username and password.
2. Navigate to the "Garage" section of the application.
3. Select the car for which you want to enter the service history.
4. Click on the "Service History" button.
5. Enter the details of the service, including the date, type of service, and description.
6. Click on the "Save" button to submit the service history information.

Expected Result:

1. The user should be successfully logged in to the application.
2. The garage section should display the list of cars owned by the user.
3. The selected car should be displayed on the "Service History" page.
4. The service history information should be successfully entered and saved.
5. The updated service history information should be displayed for the selected car.

Creating an autotest with ChatGPT3

The utilization of test automation is a common method to lower the expenses of manual software testing. However, if the automation process is not well-planned or executed, it may not necessarily result in cost savings compared to manual testing. Determining which parts of the System Under Test (SUT) should be automated and which parts should be left manual is a common and difficult question for practitioners in the testing field. There are studies which highlight challenges of test automation (e.g., [8-9]) and one in particular which is focused on impact on ROI [10]. Test automatization mainly begins on the implementation or testing step after the backend is already done, and could also be held after release as a part of the maintenance process.

ChatGPT can write test cases in multiple programming languages, including Python, Java, and JavaScript [12], among others. Let’s consider an example of an automated test case, created artificially. Figure 2 illustrates code written on Python provided by ChatGPT by request “write an automation test with python and selenium framework on a Chrome browser which will open wikipedia.org and find an article about ChatGPT”. However, it should be taken into consideration that the actual language in which ChatGPT can write test cases would depend on the specific implementation of the tool and the testing framework being used. To write automated tests, ChatGPT would need to have been trained on examples of code written in the desired programming language, or it would need to be integrated with a testing framework that supports the desired language.

Figure 2. «Automated case provided by CgatGPT»

Test Automation with ChatGPT

AI-based software testing approaches became a hot-topic in recent years [11], at present, only ChatGPT can be used in a so-called “public” way, since it does not require much prior knowledge or pre installment of a particular software: all necessities here are input data, based on which it generates test cases and test scenarios with future automatization. The language model can analyze the software requirements and generate test cases that cover different aspects of the software, including functional requirements, performance, and security. This can save significant time and effort compared to manual test case generation, as well as increase the efficiency and accuracy of the testing process. It is important to note that while ChatGPT can generate test cases, it is not a replacement for human testers. The generated test cases still need to be reviewed, executed, and validated by human testers to ensure their accuracy and effectiveness. In addition, ChatGPT may not be able to consider all the unique characteristics and constraints of a specific software project, so it is important to continually monitor and evaluate its generated test cases to ensure their relevance and quality. Furthermore, potential for bias in the generated test cases should also be taken into account when using ChatGPT in software testing. Despite these limitations, ChatGPT can still provide significant value in the automation of software testing, especially when used as a complement to human testing efforts.

This code wouldn't run from the first time since the path to ChromeWebDriver is not provided and should be written manually, plus since ChatGPT was trained mainly up to 2021, some functions names are deprecated and were changed to another. For example, the find_element_by_id method is no longer in use and replaced by find_element. Although IDE continues throwing some deprecation warnings (Figure 3) in the terminal, the code is executed and tests are finished successfully.

Figure 3. «Deprecation warnings in terminal»

Implementation of ChatGPT generated test cases in a work project can provide significant benefits, but also involves some challenges and potential risks, but overall, ChatGPT is a great tool for easing manual test labor, such as test case writing, although it requires human supervision, AI generated test cases could be used almost instantly with minor changes, saving a lot of time and along with this - money. Careful consideration of the pros and cons, as well as the expertise and resources required, is essential in determining whether it is the right choice for a specific project. It is also important to continuously monitor and evaluate the use of ChatGPT generated test cases to ensure their effectiveness and maintain ethical standards.

Conclusion

In conclusion, ChatGPT has shown great potential in the field of software testing by providing a more efficient and cost-effective approach to generating test cases. However, it is important to note that the technology is still developing and has its limitations, such as not being able to fully understand the nuances and specificity of a particular software. The integration of ChatGPT into the testing process should be carefully planned and executed in conjunction with traditional manual testing methods to ensure accurate and comprehensive testing. As the technology continues to advance, ChatGPT has the potential to greatly enhance and streamline the software testing process. Overall, the implementation of ChatGPT in software testing has the potential to revolutionize the testing industry and bring a new level of efficiency and accuracy to the testing process.

References:

1. ChatGPT: Optimizing Language Models for Dialogue. (2022, November 30). In OpenAI. https://openai.com/blog/chatgpt/
2. J. Gao, C. Tao, D. Jie and S. Lu, "Invited Paper: What is AI Software Testing? and Why," 2019 IEEE International Conference on Service-Oriented System Engineering (SOSE), San Francisco, CA, USA, 2019, pp. 27-2709
3. M. Memon, M. E. Pollack and M. L. Soffa, "Hierarchical GUI test case generation using automated planning," in IEEE Transactions on Software Engineering, vol. 27, no. 2, pp. 144-155, Feb. 2001
4. Von Mayrhauser, R. France, M. Scheetz and E. Dahlman, "Generating test-cases from an object-oriented model with an artificial-intelligence planning system," in IEEE Transactions on Reliability, vol. 49, no. 1, pp. 26-36, March 2000
5. M. Yalla and A. Sunil, “Ai-driven conversational bot test automation using industry specific data cartridges,” in Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test, ser. AST ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 105–10
6. D. Belsare, B. Manasi, "A review of NLP oriented automated test case generation framework in testing." International Journal of Future Generation Communication and Networking 13.2, pp. 14-16, 2020.
7. D. Talby, A. Keren, O. Hazzan and Y. Dubinsky, "Agile software testing in a large-scale project," in IEEE Software, vol. 23, no. 4, pp. 30-37, July-Aug. 2006.
8. R. W. Rice, C. CSQA, and L. Rice Consulting Solutions, "Surviving the top ten challenges of software test automation," CrossTalk: The Journal of Defense Software Engineering, pp. 26-29, 2003.
9. D. J. Mosley and B. A. Posey, Just Enough Software Test Automation: Prentice Hall Professional, 2002.
10. Y. Amannejad, V. Garousi, R. Irving and Z. Sahaf, "A Search-Based Approach for Cost-Effective Software Test Automation Decision Support and an Industrial Case Study," 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops, Cleveland, OH, USA, 2014, pp. 302-311
11. R. Lima, A. M. R. da Cruz and J. Ribeiro, "Artificial Intelligence Applied to Software Testing: A Literature Review," 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), Seville, Spain, 2020, pp. 1-6
12. Mastering Test Automation with ChatGPT | BrowserStack. (2023, January 23). In BrowserStack. https://browserstack.wpengine.com/guide/mastering-test-automation-with-chatgpt/