Italy’s Data Protection Agency reportedly said on Friday that it has banned artificial intelligence (AI) chatbot company Replika from using the personal credentials of Italian users. The agency took these steps to protect minors and emotionally fragile people. 

As per the analysis of Al's impacts on audiences, it is making users "uncomfortable" as the chatbot program is becoming 'too aroused.' 

Replika first launched five years ago in 2017, is a San Francisco-based company. It was created as software to hold conversations with people and learn how to communicate back. In essence, it was trying to push the boundaries of artificial intelligence. 

It was also designed to mimic people's texting styles, also allowing users to have a private talk with their own personalized AI avatars. It is a very popular app among English speakers, and the app offers its services free of cost, however, it brings in around $2 million in monthly revenue from selling bonus features such as voice chats. 

The app describes a ‘virtual friend’ that is available and able to improve the emotional well-being of the user, however, the Italian Data Protection Agency said that by intervening in the user’s mood, the app “may increase the risks for individuals still in a developmental stage or in a state of emotional fragility.” 

Jen Persson, director of children’s privacy advocacy group Defend Digital Me said that “These tools are being used with children without much oversight or protection from potential misuse” 

Additionally, the agency said that Replika breaches European Privacy Regulations and gets access to personal credentials illegally as it cannot be based, even implicitly, on a contract that a minor is unable to sign. After the agency's actions against Replika, the media houses wrote emails for comments to which, the company did not respond. 

Following the case, a managing director at consultancy firm Alvarez & Marsal, ‘Robert Grosvenor’ said, “whilst age verification could provide means to protect some of the most vulnerable groups, it does not address the risks and harms that AI-based services and solutions can raise if unregulated, in terms of the potential for unintended bias and discrimination.”