The German information security agency warns of the dangers of attacks on artificial intelligence systems
The German Federal Office for Information Security (Bundesamt für Sicherheit in der Informationstechnik, or BSI) published a report this month on the dangers of how AI can be influenced by altering the data it relies on to build its models.
The report describes three possible attack methods: evasion attacks, information extraction attacks, and poisoning and backdoor attacks.
An evasion attack occurs when a threat actor introduces “malicious input” during the inference phase of an AI’s machine learning model. This involves introducing an amount of perturbation into the input, so that the model can no longer accurately predict what it is seeing, for example. This disturbance might well be visible to the human eye, but AI cannot perceive it.
Evasion attacks can also occur when AI “student” models are built from pre-existing “teacher” models. Any attack on the teaching model could be transmitted to the student and, in cases where the teaching model is publicly available and widely used, this could be particularly damaging.
Information attacks, also known as reconstruction or privacy attacks, involve rebuilding an AI model from its training data. A threatening actor could attempt to steal a model by reconstructing it based on the answers given by the original, for example, which in turn feed back to the adversary’s own model.
Other information attacks include membership inference attacks, in which a threat actor may attempt to build a new model based on differences between the AI training data and newly input data, and attribute inference attacks, where A threat actor uses publicly available data to derive an AI model. to infer private data, such as an address.
Poisoning and backdoor attacks work by targeting the data on which a model is based, flipping the label of an input to muddy the final result of a query, or directly creating a set of triggers within a data set to produce a specific result.
“An attack is successful when a backdoor model behaves normally when encountering benign images but predicts the adversary’s chosen label when presented with activated images.” the report said.
All of these attacks depend on some access to the data set on which the model is based, making them non-trivial to execute, but the possibility of damaging results is enough for BSI to feel that these threats are absolutely essential for the developers understand and know how to do it. to defend himself.
But direct attacks aren’t the only thing AI developers need to consider, according to the report.
“Apart from malicious attacks on machine learning models, a lack of understanding of their decision-making process represents a threat,” the report says. “Models could be learning spurious correlations from faulty or insufficient training data.”
“Therefore, it is useful to understand their decision process before implementing them in real-world use cases.”
