CHATBOT ASSISTANTS: IMPLEMENTATION AND ANALYSIS OF THE EFFICIENCY

CHATBOT ASSISTANTS: IMPLEMENTATION AND ANALYSIS OF THE EFFICIENCY

Daniyar Kyzyrov

Master student, International Information Technologies University,

 Kazakhstan, Almaty

Saya Sapakova

Cand. of ph. and math. sc., Associate Professor of the Department of Computer Engineering, International Information Technologies University,

Kazakhstan, Almaty

ABSTRACT

Nowadays, the utilization of chatbots is limited to only a few websites. Given that individuals cannot always adapt to the interface of every website, there is a need for an assistant to facilitate this process. Chatbots have the potential to serve various purposes and address different issues on any given website. One prominent concern on the internet is the infringement of copyright, wherein fraudulent websites imitate official sources and disguise malicious links as legitimate ones, often deceiving unsuspecting users. In this regard, chatbots can be employed as a means to propagate these scam links, allowing scammers to illicitly access sensitive user information such as personal data, social media accounts, and financial details. Hence, it is crucial to thoroughly examine chatbots and ensure their proper and effective use. Chatbots are versatile tools that can be applied in numerous domains, ranging from simple information retrieval to fields like education and medicine. This article aims to present various types of chatbots, along with their potential applications and specific uses. Furthermore, it seeks to analyze the efficacy of chatbots and propose methodologies for evaluating their effectiveness. The article will explore the origins of chatbots and highlight contemporary trends. Additionally, it will introduce a functional chatbot model designed for the Discord platform.

Keywords: Conversational Agent, Machine Learning, Natural Language Processing, Neural Network, Chatbot.

Introduction

With the surge in web usage due to the COVID-19 pandemic and the proliferation of online services, there has been a significant increase in the number of individuals engaged in web surfing. Consequently, the efficacy of fraudulent chatbot assistants has reached unprecedented levels. Scammers have become increasingly active in creating counterfeit web pages that masquerade as legitimate sources. Exploiting chatbots for fraudulent purposes has proven highly effective, as victims often fail to detect subtle alterations made to letters or symbols, which are intended to camouflage suspicious sites. For instance, within the film industry, fake chatbot assistants may provide incorrect links to specific movies. Although these links may differ from official sources, they often go unnoticed by the average user.

Social networks, such as Vkontakte, predominantly host chatbot assistants. The number of users interacting with a single chatbot on Vkontakte currently ranges from ten thousand to 1.5 million. Such a vast user base presents an opportunity for both entertainment providers and fraudsters alike. If the issue of fraudulent chatbots remains unresolved, people may lose trust even in legitimate entertainment distributors.

To mitigate problems associated with scammers and counterfeit chatbot assistants, developers should establish direct communication channels with official sources when creating chatbot assistants. By associating chatbot assistants exclusively with official sources, the risk of third-party fraud can be eliminated, ensuring that chatbots only provide official links to movies and TV shows.

Regarding chatbot development itself, there are two approaches: creating a chatbot from scratch by a developer or utilizing specialized tools designed for bot creation. These tools not only aid developers but also facilitate the entry of individuals without prior development experience into the field.

Chatbot assistants can be implemented across various messaging platforms and social networks. By adhering to official channels and eliminating suspicious chatbots, instances of fraud can be reduced, benefiting everyone involved.

Related work

Chatbots are utilized globally by individuals with diverse motivations. Analyzing the graphical representation, it is evident that the term chatbot emerged as early as 1966 [2] and gained significant popularity after the 2000s, as depicted in Figure 1. The concept of enabling communication between computers and humans originated in Alan Turing’s mind during the 1950s, leading to the development of the first chatbot named ELIZA [3].

Figure 1. Search Results from Scopus (Chatbot keyword by Year)

The usage of the keyword chatbot varies across countries, with the highest usage observed in the United States and the lowest rating in the Netherlands, as illustrated in Figure 2.

Figure 2. Search Results from Scopus (Chatbot keyword by Country)

Chatbots are versatile product that encompasses various types and applications. Table 1 demonstrates their ability to engage users through text, photos, or voice, offering distinct objectives, permissions, and services [5].

Table 1.

Chatbot Classification

Numerous reasons for utilizing chatbots have been identified [6] based on an analysis of articles about chatbot technology. Chatbots for productivity: One of the primary motivations for their usage is the simplicity, convenience, and efficiency they provide. Chatbots enhance user productivity by granting access to a wealth of information. Users appreciate chatbots when they streamline the process of obtaining information, saving them valuable time. This can be exemplified through efficient assistance in customer support scenarios or by directing users to user-friendly manuals or FAQs [7].

Chatbots as automated responders: In certain cases, individuals employ chatbots as automated responders. When someone sends them a message, people configure chatbots to provide answers to basic questions. This enables users to avoid investing time in responding to straightforward inquiries, utilizing chatbots instead.

To seek assistance and gather information: As previously mentioned, the availability and accessibility of information serve as a driving factor for utilizing chatbots.

Some individuals employ chatbots as valuable tools in their everyday online activities. For instance, when someone enjoys traveling and requires recommendations or advice, a chatbot can effortlessly provide suggestions and useful guidance for interesting destinations.

Moreover, certain individuals prefer chatbots as assistants over real people due to shyness or discomfort when interacting with others. Chatbots offer a judgment-free environment, allowing users to seek answers to even the simplest of questions without feeling uneasy.

Chatbots for entertainment: Some people engage with chatbots purely for amusement, enjoying casual conversations and relishing their responses. They pose various questions to chatbots and derive entertainment from their humorous replies. However, it’s worth noting that some individuals misuse chatbots for idle time-wasting purposes.

Chatbots for social and relational purposes: Chatbots can serve as engaging conversationalists, with certain models designed specifically for interactive communication. They gather information from user messages and employ machine learning techniques to enhance their conversational abilities, capable of addressing a wide range of inquiries.

A chatbot can assume diverse roles and functions, such as acting as a salesperson on a website. It can facilitate sales services, generate invoices, process payments, track order statuses, and fulfill client requests through integration with relevant APIs. Following integration, the chatbot can deliver results in the desired format. Additionally, chatbots can serve as information providers, and assistants, or automate website processes by presenting rules, introducing new participants, or promoting products/services (see Fig. 3).

Figure 3. Chatbot Roles

Chatbots encompass various technological approaches, with two prominent types being Rules-Based and AI-Based [8]. Rules-Based chatbots operate within predefined guidelines, as depicted in Fig. 4. During their initial development, specific scenarios are established, and chatbots strictly adhere to these programmed instructions. They are confined to operating solely within the parameters set by the developer and cannot deviate from them. Consequently, Rules-Based chatbots are unable to respond to questions or queries outside of their designated scenarios.

Figure 4. Rules-Based Chatbots

AI chatbots undergo initial training by developers, followed by the capability to learn autonomously. These chatbots exhibit more advanced functionality compared to rules-based counterparts. As illustrated in Fig. 5, AI chatbots possess the ability to comprehend user responses and requests. However, in cases where users express dissatisfaction with the response or the requested information is not found within the predefined intents, incorrect answers may be provided. To address this, a self-feeding system [9], encompassing the chatbot’s Artificial Intelligence [10], can be employed. This system enables the chatbot to generate responses based on user feedback [11], thereby improving the accuracy of future interactions.

Figure 5. AI Chatbots

As mentioned previously, chatbots can be developed either from scratch or by utilizing chatbot builders. These builders serve as development platforms that enable users to create, modify, and customize chatbots according to their specific requirements. One notable advantage of chatbot builders is that they often do not necessitate programming expertise, making them highly accessible for users of all backgrounds. Furthermore, chatbot builders typically offer integration capabilities with email systems, customer relationship management (CRM) software, and other automation tools. This integration facilitates enhanced efficiency in sales, marketing, and various other processes. Additionally, chatbot builders often provide pre-designed templates that can be customized, enabling users to swiftly create their chatbots. In terms of effectiveness, chatbots equipped with built-in machine learning and artificial intelligence exhibit notable advantages. These advanced capabilities allow them to continually improve and refine their performance over time, enhancing their overall effectiveness.

Methods and Materials

Effective chatbots can be defined as those that closely resemble human interactions. Such chatbots provide responses based on genuine messages, deliver clear and detailed answers, and are capable of fulfilling various roles on a website. However, it is important to acknowledge that any mistakes made by these chatbots can significantly reduce their efficiency. Users’ expectations are high, and any shortcomings may lead to noticeable disappointment.

One way to assess effectiveness is by examining the level of user engagement. This involves measuring the percentage of users who interacted with the chatbot out of the total number of visitors to the website or application. However, it is worth noting that this metric may not be applicable to call centers or other user service scenarios.

Another crucial metric is the number of active users. These are users who proactively initiate conversations with the chatbot, rather than merely responding to its prompts. This metric indicates the bot’s utility and demonstrates the significance of the chatbot service to users. When users independently reach out to the chatbot, it signifies its importance in assisting them.

The number of repeat users is also a significant metric to consider. While the previous metrics effectively measure chatbot efficiency, there are instances where users may only interact with the chatbot once. While this can be positive, indicating that their query was resolved, if the goal is to encourage repeated and regular contact with the chatbot, a decline in this metric suggests a need to reevaluate the chatbot’s structure.

If a chatbot is utilized for sending messages, another valuable metric to consider is the Number of Messages Read. This metric allows you to track the readability of your messages and assess the effectiveness of notifications regarding promotions, discounts, and other updates. A desirable range for the average readability of a good chatbot is typically between 40% and 70%.

Average Session Duration is another metric that can provide valuable insights, although its interpretation may vary depending on the chatbot’s purpose. For chatbots that provide services or act as assistants, shorter session durations are preferred. These chatbots should respond promptly and clearly to user messages. Conversely, for chatbots designed for entertainment purposes, longer session durations are desirable. The longer users engage with an entertaining chatbot, the more effectively it fulfills its role.

It’s important to acknowledge that chatbots may occasionally fail to comprehend user inputs or provide incorrect answers. However, these errors can serve as valuable data for further chatbot development. By leveraging the insights gained from these errors, you can enhance the effectiveness of the chatbot and improve its performance.

Goal Compliance Rate is a unique metric that measures the achievement of user-defined or chatbot creator-defined goals. It involves tracking the number of completed requests and requests that were abandoned midway, providing valuable information about goal attainment.

In addition to these quantitative metrics, semantic analysis of user interactions is also employed. This analysis helps uncover the emotional context of conversations, identify popular and valuable queries, and gather a wealth of other information to further enhance the chatbot’s performance.

Chatbot Implementation

The chatbot was developed specifically for the Discord platform using the Python programming language. Python was chosen as it is highly popular, effective, affordable, convenient, and well-suited for chatbot creation. It offers numerous libraries for Natural Language Processing (NLP) and machine learning, making it an ideal choice for developing an intent-based chatbot. The chatbot utilizes various Python libraries, including TensorFlow, NLTK, NumPy, and more, alongside the Discord API. It encompasses features such as moderation, server management, music playback in voice chat, and image recognition.

All chatbot functions are executed through specific commands that are configured with a designated prefix. By default, the prefix is set as the ‘!’ symbol. For example, the command “!setprefix =” allows users to modify the prefix according to their preferences. This flexibility ensures that users can differentiate between multiple bots and utilize the correct prefix when interacting with the chatbot.

The neural network model was created using the Keras library within TensorFlow. The model architecture consisted of three layers: an input layer, a hidden layer, and an output layer with softmax activation function. During compilation, the model employed the SGD optimizer and categorical cross-entropy loss function. It was trained using input and output arrays, utilizing a batch size of 5 and an epoch count of 200. Subsequently, the trained model was saved as an H5 file. This trained model can predict tags for new user inputs and generate appropriate responses accordingly.

To ensure proper functionality, three distinct models were initially created for the chatbot, each using different optimizers: SGD, Adam, and RMSprop. All three models were trained for 200 epochs with a batch size of 5. As depicted in Table 2, these models exhibited high training accuracies, but the testing accuracies varied and demonstrated lower values, suggesting potential overfitting. The model with the SGD optimizer achieved the highest accuracy, followed by the model with Adam optimizer, and finally the model with RMSprop optimizer. The first model also had the fastest training time, whereas the third model employing the RMSprop optimizer exhibited the slowest training time. The inference time showed slight differences.

Table 2.

Model Comparison

Overall, the first model seems to be the most optimal and efficient as it has the fastest training time and the highest training and testing accuracies, therefore, this model, with the SGD optimizer, became the basis for the chatbot.

Conclusion

In summary, chatbots have proven to be transformative tools across various domains such as medicine, sports, entertainment, education, and the economy. They enhance the quality of user service and expedite the process of finding information, saving users' time, money, and reducing frustration. These AI-powered assistants leverage technologies like Artificial Intelligence and Natural Language Processing to perform a wide range of tasks, from assisting with music selection to aiding in website development.

A functional chatbot was specifically developed for the Discord platform, focusing on interactive user dialogues. The chatbot relies on a neural network model with the SGD optimizer as its foundation. Additionally, it incorporates various features to optimize the capabilities of the Discord platform. Future plans involve expanding the chatbot's functionalities by integrating additional APIs and further expanding its database. These advancements will enhance the chatbot's dialogue capabilities and overall efficiency.

Furthermore, it was concluded that the most effective chatbots are those that serve as informative assistants, guides, or companions. Such chatbots have the potential to increase company revenues and reduce costs significantly. With the rapid progress in the field of artificial intelligence, it is anticipated that chatbots will continue to gain popularity and gradually integrate into various websites, serving as invaluable resources for information, entertainment, and more.

References:

1. Abu Shawar, B., Atwell, E. (2007). “Chatbots: Are they Really Useful?” University of Leeds.
2. Adamopoulou E., & Moussiades, L. (2020). “Chatbots: History, technology, and applications.” International Hellenic University.
3. Singh A., Ansari, M., Shaikh, S., Parbulkar, M. S., & Khan, T. (2021). “Intelligent Chatbot.” Maharashtra Institute of Technology.
4. Behera B. (2016). “Chappie - A Semi-automatic Intelligent Chatbot.” Vellore Institute of Technology.
5. Vakili A., & Shakery, A. (2019). “Enriching Conversation Context in Retrieval-based Chatbots.” University of Tehran.
6. Følstad A., & Brandtzaeg, P. B. (2017). “Why People Use Chatbots?” SINTEF Digital.
7. Thakkar J., Raut, P., Doshi, Y., & Parekh, K. (2018). “Erasmus-AI Chatbot.” University Rotterdam.
8. Abraham J. (2018). The Significance of Chatbots. Retrieved from Dzone Website: https://dzone.com
9. Hancock, B. (2019). “Learning from Dialogue after Deployment: Feed Yourself, Chatbot!” Computer Science Dept. Stanford University.
10. Pal A., Bhalotia, S., Rathod, V., Bisen, S., & Lalwani, T. (2018). “Implementation of a Chatbot System using AI and NLP.” K. J. Somaiya Institute of Engineering and Information Technology.
11. Ghias A.R., Guo, C., Sarikaya, R., & Ponnusamy, P. (2019). “Amazon Alexa, Feedback-Based Self-Learning in Large-Scale Conversational AI Agents.” Retrieved from website Amazon: https://www.amazon.com
12. Farhan L., Kaiwartya, O., & Kharel, R. (2018). “A Concise Review on Internet of Things (IoT) - Problems, Challenges and Opportunities.” University of Oulu.
13. Caldarini G., Jaf, S., & McGarry, K. (2021). “A Literature Survey of Recent Advances in Chatbots.”
14. Brain [Brn.Ai] Code for Equity (2021). “Chatbot Trends Report 2021.” Retrieved  from chatbotsjournal website: https://chatbotsjournal.com/chatbot-trends-report-2021-b15479c404e4
15. Khosravi H., Shafie, M.R., Hajiabadi, M., Raihan, A. S., & Ahmed, I. (2021). “Chatbots and ChatGPT: A bibliometric analysis and systematic review of publications in Web of Science and Scopus databases.” Journal of Information Science, 47(5), 550-573.
16. Benlian A., Adam, M., & Wessel, M. (2020). “AI-based chatbots in customer service and their effects on user compliance.” Technical University of Munich.