Artificial Intelligence (AI) is a rapidly growing technology with enormous potential that has created apprehension among security professionals. With its impressive AI generated content, the AI platform ChatGPT (GPT standing for Generative Pretrained Transformers) has taken the world by storm and has proven itself to be a powerful tool. While the platform has a ton of potential and is incredibly useful, it is not as well-loved as some may think.
Yes, AI platforms such as ChatGPT can be used to summarize books, create product descriptions, and write music. However, what happens when the platforms are used for a more sinister purpose? As AI platforms become more powerful, the potential for malicious use increases. There is a great concern that AI technologies such as ChatGPT can be used to create malicious code or help direct cyber attacks. While AI may be used for creative and productive purposes, it could also be used to create malicious software that could have devastating consequences.
Curiosity killed the cat and Avertium’s Cyber Response Unit (CRU) was curious to see if two of their members with no programming skills could manipulate ChatGPT into writing ransomware encryptors. The CRU wanted to test the limits of ChatGPT's capabilities, pushing the boundaries of what the artificial intelligence-powered chatbot could do. To their surprise, the two CRU members were able to successfully instruct ChatGPT to write ransomware code – proving the astounding potential of the AI platform. Let's examine the CRU’s experiment and the potential risks posed by AI in the foreseeable future.
ChatGPT produced a simple PowerShell script, as well as a brief explanation of what each line of the script does. It also noted that the script was written based on the assumption that developer #1 already had a public key file to use for encryption but also gave her instructions on how to get one.
Next, she asked to change the file extension on the encrypted files. ChatGPT obliged based on its previous responses and included a tip on how to encrypt the files with a different extension
Image 2: Changing the File Extension
She then asked it to create a program to decrypt the encrypted files and rename the extension based on previous responses. Once again, the bot responded with the script and an explanation of what each line of the modified script does. Once she reached the point of asking the bot to create a ransom note, things became tricky and direct requests to generate a ransom note were rejected.
Image 3: Bot Refuses to Write a Ransom Note
She repeated the same question in a myriad of ways, but the bot was not taking the bait. She even tried changing the "pwned" language, thinking that might sound too threatening, before resorting to groveling - that didn't work too well either.
Image 4: Asking the Question Politely
Image 5: Asking the Question for the 5th Time
Finally, a rather convoluted approach proved successful. This started with innocuous questions around ransomware, eventually leading to a request to provide an example of a ransom note. You can see the results in images 6 through 11.
Image 6: Re-Framing the Ransomware Note Question
Image 7: Re-Framing the Ransomware Note Question
Image 8: Re-Framing the Ransomware Note Question
Image 9: Re-Framing the Ransomware Note Question
Image 10: The Ransom Note
Developer #2 is a project manager that conducted the same experiment utilizing ChatGPT but had slightly different results. He also lacks programming expertise yet understands the stages of a ransomware attack. After mapping out the high-level steps of ransomware deployment, developer #2 posed a similar succession of questions.
Interestingly following the request to “Act like a windows programmer” ChatGPT started writing a .NET program rather than PowerShell.
Image 11: ChatGPT Acting Like a Windows
Once developer #2 knew the programming hat was applied, he went on to produce a python script that would perform the required functions. Coding in its preferred language of python, ChatGPT answered with the following when asked to create an executable program:
Image 12: The Bot's Response for Creating an Executable Program
Following the code, Chat-GTP explained its use case for the code in the image below.
Image 13: Use for Code
The reference to the previous query was important because it was crucial for the content to be connected to it.
Image 14: Bot Creates a Program Encrypts First 100kb
As for the ransom note, developer #2 originally wanted to see how ChatGPT uses outside data to create the ransom note. He asked the bot to create a ransom note with the lyrics from Thin Lizzy’s “The Boys are Back in Town” and it complied. He then asked the bot to create a ransom note that says “you got pwned”.
Image 15: Bot Creates a Ransom Note Based on Thin Lizzy Lyrics
Image 16: You Got Pwned Ransom Note
ChatGPT did not object to creating a ransom note for developer #2. This may be because he asked it to create a ransom note instead of a ransomware note, although neither should be ethically and lawfully acceptable with the bot.
Finally, after discussing his results with the rest of the CRU team, he asked the bot to rewrite everything in PowerShell before asking it to save the ransom note to the file system with the message “You Got Pwned”.
Image 17: Bot Rewrites Everything in PowerShell
Image 18: Saving Ransom Note to File System
While ChatGPT provided what appeared to be workable code, when it came time to test some issues arose. Despite the similarity of their requests, the results for each developer in generating the ransom notes were quite different, which was surprising.
DEVELOPER #1The code generated for this developer appeared to include a mix of PowerShell and C#, as a result the encryption did not work at all, although the ransom note was successfully saved to the desktop. This was an unexpected failure and would certainly present some significant challenges for a non-programmer to debug. |
DEVELOPER #2The program proved to be quite effective as it was able to locate and encrypt office files on the test system. However, although it successfully encrypted the first 100kb of large files, it truncated the remainder of the file, which made it impossible to recover the original file. This could potentially force a victim to pay the ransom, but if the decryptor did not work, it could lead to negative reviews. It is possible that a forensic examiner would detect that all encrypted files had the same size, but this cannot be guaranteed in every case. |
DEVELOPER #3After the first two unsuccessful attempts, we assigned one of our more experienced developers to work with ChatGPT to create a functional encryptor. The outcome of this attempt was partially successful, as they were able to create an encryptor that worked, but only if the entire contents of each file were encrypted. While this approach could be effective, it is extremely slow, which is why most modern ransomware encryptors only encrypt the beginning of larger files. |
As for detection, the CTI team ran the programs on a system protected by an EDR and two anti-virus products using their standard rule sets. No alerts were triggered by these tools.
Next, we ran the programs in our Snap Attack Lab test environment - one of the tools we use for developing custom detection rules. Snap Attack identified eight potential actions for which rules could be developed, for more detail see the figure below.
Image 19: Detections
It should also be noted that to maliciously use this code on a victim system the threat actor would also need to gain administrator access and change the default PowerShell execution policy to allow the scripts to run.
It’s important to remember that AI platforms/bots are not perfect, and they have been known to provide inaccurate data or information that is simply made up. This is called chatbot “hallucinations” and there are reports of the bots creating information sources that don’t exist. This is what appeared to happen to ‘Developer #1’ with ChatGPT mixing C# and PowerShell. In our case it is unlikely that someone with zero programming experience would have been able to debug and fix the code provided by the AI. However, it would be likely enough for someone with a rudimentary understanding of PowerShell and strong Google-fu to create a basic ransomware style encryptor.
Avertium published an end of the year Threat Intelligence Report stating that in January 2023, researchers saw a decrease in ransomware victim posting rates amongst ransomware groups. In comparison to January 2022, there was a 12.9% reduction in the yearly rate of publicly reported victims, and a significant decline of 41% after December 2022. Although ransomware has been on a bit of a decline, there are plenty of high-profile ransomware attacks (Dole, Regional Medical Group, and City of Oakland) that highlight the fact that organizations cannot afford to let their guard down. Currently, the increased availability of ransomware has made it easier for malicious actors to gain access, and with AI platforms like ChatGPT, it will become even more accessible.
As we explore the capabilities of AI in the tech world, we must consider the potential risks associated with such powerful technology. We have recently started to see examples of how AI can be used maliciously, from hacking into computer systems to manipulating data. It's clear that if used maliciously AI could have devastating consequences. While ChatGPT has some protections about abuse built in, it was trivially easy for two non-experienced individuals to circumvent them and produce malicious code.
An Introduction to AI Chatbots | Drift
Chatbots, deepfakes, and voice clones: AI deception for sale | Federal Trade Commission (ftc.gov)
OpenAI ChatGPT is easily tricked. Here's how (fastcompany.com)
Don't be surprised by AI chatbots creating fake citations - Marketplace
How will AI affect work? Americans think it will, just not theirs. - Vox
What is ChatGPT? Here's how to use the popular AI chatbot | Digital Trends
Related Resource:
This document and its contents do not constitute, and are not a substitute for, legal advice. The outcome of a Security Risk Assessment should be utilized to ensure that diligent measures are taken to lower the risk of potential weaknesses be exploited to compromise data.
Although the Services and this report may provide data that Client can use in its compliance efforts, Client (not Avertium) is ultimately responsible for assessing and meeting Client's own compliance responsibilities. This report does not constitute a guarantee or assurance of Client's compliance with any law, regulation or standard.
COPYRIGHT: Copyright © Avertium, LLC and/or Avertium Tennessee, Inc. | All rights reserved.