Asymmetries in Adversarial Settings

Even without formal guarantees of their effectiveness, adversarial attacks against Machine Learning models frequently fool new defenses. We identify six key asymmetries that contribute to this phenomenon and formulate four guidelines to build future-proof defenses by preventing such asymmetries. We also prove that attacking a classifier is NP-complete, while defending from such attacks is Sigma_2^P-complete. We then introduce Counter-Attack (CA), an asymmetry-free metadefense that determines whether a model is robust on a given input by estimating its distance from the decision boundary. Under specific assumptions CA can provide theoretical detection guarantees. Additionally, we prove that while CA is NP-complete, fooling CA is Sigma_2^P-complete. Even when using heuristic relaxations, we show that our method can reliably identify non-robust points. As part of our experimental evaluation, we introduce UG100, a new dataset obtained by applying a provably optimal attack to six limited-scale networks (three for MNIST and three for CIFAR10), each trained in three different manners.