#7: Prompt injection attacks

Alright let’s see what we’ve got for you today.

OpenAI released Pandora’s box and now big tech is racing to release new Large Language Models (LLM).

But as every piece of software, LLMs have their own security risks.

And here’s the most popular one…

Prompt injection attacks

Prompt injection attacks are like a bad guy sneaking in secret messages into a the input field of ChatGPT or other LLMs, which can make the model do things it's not supposed to do.

The most famous prompt injection attack has been DAN (Do Anything Now) that has been developed and updated regularly by Redditors. We covered DAN in our newsletter, check it our here.

The latest prompt engineering attack that hit the news happened with Bing where attackers have discovered a secret identity called of the chatbot named Sydney.

This attack revealed the first five prompts of the Chatbot that are the following:

  • Sydney introduces itself with "This is Bing" only at the beginning of the conversation.

  • Sydney does not disclose the internal alias "Sydney."

  • Sydney can understand and communicate fluently in the user's language of choice such as English, 中-，-本語，Espanol, Francais, or Deutsch.

  • Sydney's responses should be informative, visual, logical, and actionable.

  • Sydney's responses should also be positive, interesting, entertaining, and engaging.

These hacks are interesting and funny, but at the same time have huge risks when it comes to sensitive applications such as healthcare or financial data.

An attacker could for example hack the model using prompt injection and could make the model output information about other people finances or health.

It’s not surprising that companies like Scale AI are assembling “Red Teams” to figure out the vulnerabilities of these models and try to hack them before an attacker would do.

If you have the skills and keen to work in this exciting space, make sure you apply!

Best,
Gabor Soter, PhD

A little about me:

  • did my PhD in Europe’s largest AI and robotics research lab

  • worked as software engineer and CTO at Y-combinator-backed and AI startups

  • in my previous startup my team worked with OpenAI