In Feb 2024, a finance manager Ravi (name anonymized) received a message from his CFO. The message said their company needed to make a secret payment, and asked Ravi to join a video call to discuss details. Ravi was initially suspicious. But, he put aside his early doubts when he joined the video call and saw his CFO and other colleagues on the call. He followed the CFO’s ask and authorized a payment of $25 million to the account provided.

It turns out that all the faces and voices on the video call were AI-generated deepfakes. This is a real incident from Hong Kong.

We live in interesting times. AI is having a wide-ranging impact on the planet, both good and bad. It enables amazing new use cases. It is displacing jobs. But most importantly it is challenging the foundational element that made society possible – trust. This article covers one slice of that topic, AI’s impact on cybersecurity

Artificial Intelligence 101

ChatGPT has caught the world’s fancy in the last two years. There is much more to AI than ChatGPT. Here is a quick flyby, enough to appear informed at a social event!

Artificial Intelligence refers to the ability of a computer to perform tasks “commonly associated” with humans. There are, broadly speaking, two ways in which AI may be applied.

• Predictive AI is used when the goal is to pick the best choice (according to a given scoring method) from a pre-determined range of choices. Examples: Deepblue from IBM beat then-champion Garry Kasparov, using predictive AI that could analyze 200 million moves in one second to pick the best move. A self-driving car picks which way and how much to steer based on what it senses in the environment. Netflix recommends the shows you are most likely to enjoy. Google, Meta, and Bing rank the ads most relevant to you.

• Generative AI is used when the goal is to generate new content. Large Language Models (LLMs) are the most popular cases for Generative AI. They are designed to generate text. ChatGPT is the most famous application for LLMs.

The way AI systems learn has greatly evolved. In the early days, humans programmed the system based on their knowledge. In the 1990s, machine learning picked up. In this approach, you first train an AI system by feeding it large amounts of input-output pairs and letting it find patterns. Once trained, the AI system, aka model, can predict the output for any new inputs you give it. Deep learning builds on this by using multiple layers of such a system. All well-known AI systems today learn via deep learning.

Fun fact: Machine learning and deep learning use Linear Algebra, which every student at IITB learns in the first year.

All AI systems that exist today are what academicians refer to as Artificial Narrow Intelligence, meaning they are designed to solve a narrow use case, and are trained by humans for that use case. Researchers are working towards Artificial General Intelligence (AGI), which is AI that can learn to do new tasks without humans training it. The holy grail of researchers is Artificial Super Intelligence, which is AI that surpasses most humans in cognitive abilities. This is still science fiction, despite what all the sensationalists may want us to believe.

Now that we understand the types of AI, let’s understand the good, bad, and ugly of how AI impacts cybersecurity.

AI helps malicious actors

Let’s start with the bad. With every technology wave – internet, mobile, cloud – there has been an initial phase where malicious actors embrace the new technology much faster than good citizens do. AI is no different.

• As described in the real story at the beginning of this article, malicious actors are already using Generative AI to fool users with deep fakes and realistic phishing mails. It takes only 20 seconds of a person’s voice and about 500 photos of the person to create very convincing deepfakes. Limited deepfakes, good enough for video calls and fun applications, require even fewer photos and voice clips. Elections across the world have been a prime target of deepfakes due to the stakes involved, followed by tech support scams.
• Malicious actors are using LLMs to write and evolve malware faster than ever, which is a challenge for anti-malware products.
• Bots on social media have been problematic for many years. E.g. during the US election of 2016, there was a heavy amount of propaganda on both Twitter and Facebook generated by bots. Now with Generative AI, bots are becoming good even at interaction.
• Once attackers get a foothold in a business, they can leverage LLMs to find their way around the business much faster. E.g. Let’s say an employee in a company clicked on a malicious URL and got malware on their machine. This gave the attacker a foothold in the company. From here they can access all documents that the employee has access to. Instead of spending days reading through them, they can now use LLMs to summarize the documents instantly to understand the people and assets in the rest of the company. They can even use LLMs to instantly draft code to automate the next steps of the attack. This would have taken days in the past. The attacker thus moves at high speed, which creates a challenge for defenders.

AI helps defenders

The good news is that AI helps strengthen cybersecurity also. It is helping defenders run fast (ideally faster than attackers), cover more territory, and also directly combat the issues mentioned above.

Predictive AI has been used in cybersecurity for many years. It is impossible for a cybersecurity team in a business to manually keep a watch over every activity of every employee and machine. Instead it is common to use AI for this. Every computer and every application generates logs of its activity. If an AI model can be trained with many months of “good” logs (meaning before there is a cyberattack) then such a trained model can in the future detect a deviation from that baseline and raise an alarm. The deviation may be a sign of a cyberattack.

Some security products such as Microsoft’s Defender and Abnormal Security, use predictive AI models that are trained with samples of good emails and phishing emails. With this training they can churn through millions of emails fast and identify phishing emails with pretty good accuracy.

Generative AI is a new branch of AI, but it is also picking up momentum in cybersecurity. This is an area of active research. Early successes include natural language interfaces on top of existing security tools, to make it easy for defenders to use those tools. An extension to this is co-pilots, which allow users to type higher level instructions (e.g. “find impact of this vulnerabilty”) and let the AI break it down into tasks. A number of companies such as https://Simbian.AI are working on advanced use cases of Gen AI such as

• investigating security alerts (understanding alerts raised by various security tools and investigating the impact),
• threat hunting (using an understanding of attackers’ techniques to find them in your network), and
• threat modeling (understanding a software design and finding ways to exploit its vulnerabilities).

In addition to speeding up defenders in their usual tasks, many companies and researchers are also working on ways to directly combat how attackers are using AI, such as deepfakes. One example is the wonderful work done by https://TrueMedia.org, a non-profit organization.

The hope is that defenders embrace AI fast enough to offset how fast attackers are leveraging AI.

AI needs to be secured too!

As we rush to use AI for every use case including cybersecurity, it is important to remember that every computer system comes with security flaws. AI is no exception.

The majority of users of AI cannot directly address this problem. When you buy a product that uses AI, read reviews and ask the vendor what they do to address the problem.

OWASP, a non-profit organization for security standards, maintains a popularly used list of vulnerability types. A few worth mentioning here are:

• Model vulnerabilities. If an attacker can reverse engineer what inputs lead to what outputs, then they can game the inputs to force certain (bad) outputs.
• Training vulnerabilities. A model is only as good as its training data. You may have heard in the news every so often of bots going unhinged because they learned bad language or incorrect facts from pranksters and malicious users.
• Prompt injection. This applies specifically to Generative AI models. An attacker can manipulate the inputs to a model to produce a desired output such as a misleading or harmful content.
• Hallucination. The job of Gen AI models is to generate new content. When this generates content that is not consistent with (undisputed) facts, it is known as hallucination. A simple example is shown below with ChatGPT. Left unchecked, hallucination can result in users being misled to dangerous decisions.

Example of Hallucination in ChatGPT

Call to action

AI is here to stay. Generative AI in particular is finding its way into every walk of life. Very soon, those who know how to leverage AI to their benefit will be at a distinct advantage over those who have not embraced AI. I highly recommend that everyone spend at least 30 mins every day using an AI product consciously. e.g. use ChatGPT to draft documents, try different prompts and observe how it behaves. Test the limits of what works and what fails.

If you wish to dive deeper into the AI technology itself, a great starting resource is https://deeplearning.ai

Arin Barde is the son of our alumnus, Sumedh Barde, and he is 18 years old. Fundamatics team sincerely appreciates Arin’s valuable contribution and wishes him every success on his promising journey in the field of cybersecurity. With his passion for cybersecurity, we can’t wait to see him unlock new doors in the digital world.