index.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>secML</title>
    <link>https://secml.github.io/index.xml</link>
    <description>Recent content on secML</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Fri, 23 Mar 2018 00:00:00 +0000</lastBuildDate>
    <atom:link href="https://secml.github.io/index.xml" rel="self" type="application/rss+xml" />
    
    <item>
      <title>Class 8: Testing of Deep Networks</title>
      <link>https://secml.github.io/class8/</link>
      <pubDate>Fri, 23 Mar 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class8/</guid>
      <description>

&lt;h2 id=&#34;deepxplore-automated-whitebox-testing-of-deep-learning-systems&#34;&gt;DeepXplore: Automated Whitebox Testing of Deep Learning Systems&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana. 2017. &lt;em&gt;DeepXplore: Automated Whitebox Testing of Deep Learning Systems&lt;/em&gt;. In Proceedings of ACM Symposium on Operating Systems Principles (SOSP ’17). ACM, New York, NY, USA, 18 pages. &lt;a href=&#34;https://arxiv.org/pdf/1705.06640.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;As deep learning is increasingly applied to security-critical domains, having high confidence in the accuracy of a model&amp;rsquo;s predictions is vital.  Just as in traditional software development, confidence in the correctness of a model&amp;rsquo;s behavior stems from rigorous testing across a wide variety of possible scenarios.  However, unlike in traditional software development, the logic of deep learning systems is learned through the training process, which opens the door to many possible causes of unexpected behavior, like biases in the training data, overfitting, underfitting, etc.  As this logic does not exist as an actual line of code, deep learning models are extremely difficult to test, and those who do are are faced with two key challenges:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;How can all (or at least most) of the model&amp;rsquo;s logic be triggered so as to discover incorrect behavior?&lt;/li&gt;
&lt;li&gt;How can such incorrect behavior be identified without manual inspection?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;To address these challenges, the authors of this paper first introduce &lt;em&gt;neuron coverage&lt;/em&gt; as a measure of how much of a model&amp;rsquo;s logic is activated by the test cases. To avoid manually inspecting output behavior for correctness, other DL systems designed for the same purpose are compared across the same set of test inputs, following the logic that if the models disagree than at least one model&amp;rsquo;s output must be incorrect.  These two solutions are then reformulated into a joint optimization problem, which is implemented in the whitebox DL-testing framework DeepXplore.&lt;/p&gt;

&lt;h3 id=&#34;limitations-of-current-testing&#34;&gt;Limitations of Current Testing&lt;/h3&gt;

&lt;p&gt;The motivation for the DeepXplore framework is the inability of current methods to thoroughly test deep neural networks.  Most existing techniques to identify incorrect behavior require human effort to manually label samples with the correct output, which quickly becomes prohibitively expensive for large datasets.  Additionally, the input space of these models is so large that test inputs cover only a small fraction of cases, leaving many corner cases untested.  Recent work has shown that these untested cases near model decision boundaries leave DNNs vulnerable to adversarial evasion attacks, in which small perturbations to the input cause a misclassification.  And even when these adversarial examples are used to retrain the model and improve accuracy, they still do not have enough model coverage to prevent future evasion attacks.&lt;/p&gt;

&lt;h3 id=&#34;neuron-coverage&#34;&gt;Neuron Coverage&lt;/h3&gt;

&lt;p&gt;To measure the area of the input space covered by tests, the authors define what they call &amp;ldquo;neuron coverage,&amp;rdquo; a metric analogous to code coverage in traditional software testing.  As seen in the figure below, neuron coverage measures the percentage of nodes a given test input activates in the DNN, analogous to the percentage of the source code executed on code coverage metrics.  This is believed to be a better measure of the robustness test inputs because the logic of a DNN is learned, not programmed, and exists primarily in the layers of nodes that compose the model, not the source code.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/neuron_coverage.png&#34; width=&#34;500&#34;&gt;
&lt;div class=&#34;caption&#34;&gt; &lt;a href=&#34;(https://arxiv.org/pdf/1705.06640.pdf)&#34;&gt;&lt;em&gt;Source&lt;/em&gt;&lt;/a&gt; &lt;/div&gt;
&lt;/p&gt;

&lt;h3 id=&#34;cross-referencing-oracles&#34;&gt;Cross-referencing Oracles&lt;/h3&gt;

&lt;p&gt;To eliminate the need for expensive human effort to check output correctness, multiple DL models are tested on the same inputs and their behavior compared.  If different DNNs designed for the same application produce different outputs on the same input, at least one of them should be incorrect, therefore identifying potential model inaccuracies.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/oracle.png&#34; width=&#34;500&#34;&gt;
&lt;div class=&#34;caption&#34;&gt; &lt;a href=&#34;(https://arxiv.org/pdf/1705.06640.pdf)&#34;&gt;&lt;em&gt;Source&lt;/em&gt;&lt;/a&gt; &lt;/div&gt;
&lt;/p&gt;

&lt;h3 id=&#34;method&#34;&gt;Method&lt;/h3&gt;

&lt;p&gt;The primary objective for the test generation process is to maximize the neuron coverage and differential behaviors observed across models.  This is formulated as a joint optimization problem with domain-specific constraints (i.e., to ensure that a test case discovered is still a valid input), which is then solved using a gradient ascent algorithm.&lt;/p&gt;

&lt;h3 id=&#34;experimental-set-up&#34;&gt;Experimental Set-up&lt;/h3&gt;

&lt;p&gt;The DeepXplore framework was used to test three DNNs for each of five well-known public datasets that span multiple domains: MNIST, ImageNet, Driving, Contagio/Virustotal, and Drebin.  Because these datasets include images, video frames, PDF malware, and Android malware, different domain-specific constraints were incorporated into DeepXplore for each dataset (e.g. pixel values for images need to remain between 0 and 255, PDF malware should still be a valid PDF file, etc.).  The details of the chosen DNNs and datasets can be seen in the table below.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/datasets.png&#34; width=&#34;700&#34;&gt;
&lt;div class=&#34;caption&#34;&gt; &lt;a href=&#34;(https://arxiv.org/pdf/1705.06640.pdf)&#34;&gt;&lt;em&gt;Source&lt;/em&gt;&lt;/a&gt; &lt;/div&gt;
&lt;/p&gt;

&lt;p&gt;For each dataset, 2,000 random samples were selected as seed inputs, which were then manipulated to search for erroneous behaviors in the test DNNs.  For example, in the image datasets, the lighting conditions were modified to find inputs on which the DNNs disagreed. This is shown in the photos below from the Driving dataset for self-driving cars.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/driving.png&#34; width=&#34;400&#34;&gt;
&lt;div class=&#34;caption&#34;&gt; &lt;a href=&#34;(https://arxiv.org/pdf/1705.06640.pdf)&#34;&gt;&lt;em&gt;Source&lt;/em&gt;&lt;/a&gt; &lt;/div&gt;
&lt;/p&gt;
The top row has the original images with arrows indicating that all three DNNs agreed on the decision and the bottom row has the images with modified lighting conditions with arrows showing that at least one of the models made a difference decision than the other two.&lt;/p&gt;

&lt;h2 id=&#34;deep-k-nearest-neighbors-towards-confident-interpretable-and-robust-deep-learning&#34;&gt;Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Nicolas Papernot, Patrick McDaniel. 2018. &lt;em&gt;Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning&lt;/em&gt;. &lt;a href=&#34;https://arxiv.org/pdf/1803.04765.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2 id=&#34;k-nearest-neighbors&#34;&gt;k-Nearest Neighbors&lt;/h2&gt;

&lt;p&gt;Deep learning is ubiquitous. Deep neural networks achieve a good performance on challenging tasks like machine translation, diagnosing medical conditions, malware detection, and classification of images. In this research work, the authors mentioned about three
well-identified criticisms directly relevant to the security. They are the lack of reliable confidence in machine learning, model interpretability and robustness. Authors introduced the Deep k-Nearest Neighbors (DkNN) classification algorithm in this research work. It enforces the conformity of the predictions made by a DNN model on the test data with respect to the model’s training data. For each layer in the neural network, the DkNN performs a nearest neighbor search to find training points for which the layer’s output is closest to the layer’s output on the test input. Then they analyze the assigned label of these neighboring points to make it sure that the intermediate layer&amp;rsquo;s computations remain conformal with the final output model’s prediction.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/Dknn.png&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Intuition behind the Deep k-Nearest Neighbors (DkNN)
&lt;/p&gt; 

&lt;p&gt;Consider a deep neural network (in the left of the figure), representations output by each layer (in the middle of the figure) and the nearest neighbors found at each layer in the training data (in the right of the figure). Drawings of pandas and school buses indicate training points. We can observe that confidence is high when there is homogeneity among the nearest neighbors labels. Interpretability of the outcome of each layer is provided by the nearest neighbors. Robustness stems from detecting nonconformal predictions from nearest neighbor labels found for out-of-distribution inputs across different layers.&lt;/p&gt;

&lt;h3 id=&#34;algorithm&#34;&gt;Algorithm&lt;/h3&gt;

&lt;p&gt;The psudo-code for their k-Nearest Neighbors (DkNN) that the authors introduced in ensuring that the intermediate layer&amp;rsquo;s computations remain conformal with the respect to the final model&amp;rsquo;s prediction is given below-&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/algorithm_dknn.png&#34; width=&#34;400&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Code snippet of Deep k-Nearest Neighbor   
&lt;/p&gt; 

&lt;h2 id=&#34;basis-for-evaluation&#34;&gt;Basis for Evaluation&lt;/h2&gt;

&lt;p&gt;Basis for evaluating robustness, interpretability, Confidence are discussed below-
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/basis_evalution.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Basis for Evaluation
&lt;/p&gt;&lt;/p&gt;

&lt;h2 id=&#34;evaluation-of-confidence-credibility&#34;&gt;Evaluation of Confidence/ Credibility&lt;/h2&gt;

&lt;p&gt;In their experiments, they measured high confidence on inputs. From the experiment they observed that credibility varies across both in- and out-of-distribution samples. They tailored their evaluation to demonstrate that the credibility is well calibrated. They performed their experiments on both benign and adversarial examples.&lt;/p&gt;

&lt;h2 id=&#34;classification-accuracy&#34;&gt;Classification Accuracy&lt;/h2&gt;

&lt;p&gt;In their experiment, they used three datasets. First, hand written recognition task of MNIST dataset, SVHN dataset and the third one is GTSRB dataset. In the following figure, we can observe the comparison of the accuracy between DNN and DkNN model on three different dataset.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/classification_accuracy.png&#34; width=&#34;400&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Classification accuracy of the DNN and DkNN: the DkNN has a limited impact on or improves performance.    
&lt;/p&gt; 

&lt;h2 id=&#34;credibility-on-in-distribution-samples&#34;&gt;Credibility on in-distribution samples&lt;/h2&gt;

&lt;p&gt;Reliability diagrams are plotted for the three different datasets (MNIST, SVHN and GTSRB) below:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/evaluation_confidence_dknn.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Reliability diagrams of DNN softmax confidence
(left) and DkNN credibility (right) on test data—bars (left
axis) indicate the mean accuracy of predictions binned by
credibility; the red line (right axis) illustrates data density
across bins. The softmax outputs high confidence on most of
the data while DkNN credibility spreads across the value range.&lt;br /&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;On the left, they visualized the estimation of confidence output by the DNN softmax and it is calculated by the probability \(arg ~max_{j} ~f_j(x)\). On the right, they plotted the credibility of DkNN predictions. From the graph, it may appear that the softmax is better calibrated than the corresponding DkNN. Because its reliability diagrams are closer to the linear relation between accuracy and DNN confidence. But if the distribution of DkNN credibility values are considered then it surfaces that the softmax is almost always very confident on test data with a confidence above 0.8. DkNN uses the range of possible credibility values for datasets like SVHN (test set contains a larger number of inputs that are difficult to classify).&lt;/p&gt;

&lt;!-- ## Mislabeled Inputs

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/mislabeled.png&#34; width=&#34;600&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Mislabeled inputs from the MNIST (top) and SVHN
(bottom) test sets: we found these points by searching for
inputs that are classified with strong credibility by the DkNN
in a class that is different than the label found in the dataset.    
&lt;/p&gt;  --&gt;

&lt;h2 id=&#34;credibility-on-out-of-distribution-samples&#34;&gt;Credibility on out-of-distribution samples&lt;/h2&gt;

&lt;p&gt;Images from NotMNIST is identical to MNIST but the classes are non-overlapping. For MNIST, the first set of out-of-distribution samples contains images from the NotMNIST dataset. For SVHN, the out-of-distribution samples contains images from the CIFAR-10 dataset. Again, they have the same format but there is no overlap between SVHN and CIFAR-10. For both the MNIST and SVHN datasets, they rotated all the test inputs by an angle of 45 degree to generate a second set of out-of-distribution samples.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/dknn-on-test-data.png&#34; width=&#34;600&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; DkNN credibility vs. softmax confidence on outof-distribution
test data: the lower credibility of DkNN
predictions (solid lines) compared to the softmax confidence
(dotted lines) is desirable here because test inputs are not part
of the distribution on which the model was trained—they are
from another dataset or created by rotating inputs.    
&lt;/p&gt; 

&lt;p&gt;In the above figure, the credibility of the DkNN on the out-of-distribution samples is compared with the DNN softmax on MNIST (left) and SVHN (right). The DkNN algorithm has an average credibility of 6% and 9% to inputs from the NotMNIST and rotated MNIST test sets respectively, compared to 33% and 31% for the softmax probabilities. We find the same observation for SVHN model. Here, the DkNN assigns an average credibility of 15% and 18% to CIFAR-10 and rotated SVHN inputs, compared to 52% and 33% for the softmax probabilities.&lt;/p&gt;

&lt;h2 id=&#34;evaluation-of-the-interpretability&#34;&gt;Evaluation  of the interpretability&lt;/h2&gt;

&lt;p&gt;Here, they have considered the model being bias to the skin color of a person. In a recent study, Stock and Cisse demonstrate how an image of former US president Barack Obama throwing an American football in a stadium ResNet model. They reproduce their experiment and apply the DkNN algorithm to this model. They plotted the 10 nearest neighbors from the training data. These neighbors are computed by the last hidden layer of ResNet model.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/evaluation-interpretability.png&#34; width=&#34;600&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Debugging ResNet model biases—This illustrates how
the DkNN algorithm helps to understand a bias identified by
Stock and Cisse [105] in the ResNet model for ImageNet. The
image at the bottom of each column is the test input presented
to the DkNN. Each test input is cropped slightly differently to
include (left) or exclude (right) the football. Images shown at
the top are nearest neighbors in the predicted class according
to the representation output by the last hidden layer. This
comparison suggests that the “basketball” prediction may have
been a consequence of the ball being in the picture. Also
note how the white apparel color and general arm positions of
players often match the test image of Barack Obama.    
&lt;/p&gt; 

&lt;p&gt;On the left side of the above figure, the test image that is processed by the DNN is the same as that of the one used by Stock and Cisse. It contains 7 black and 3 white basketball players. They are similar to the color and also located in the air. They assumed that the ball play an important role in prediction. So, they ran another experiment with the same image but now cropping the image to remove the ball. Now the model predictated it as he is playing racket. Neighbor in this training class are white players. Image share certain charateristics, such as the background is green and most of the people are wearing white dresses and holding there hands in the air. In this example, besides the skin color, the position and appearance of the ball also contributed to the model&amp;rsquo;s prediction.&lt;/p&gt;

&lt;h2 id=&#34;evaluation-of-robustness&#34;&gt;Evaluation of Robustness&lt;/h2&gt;

&lt;p&gt;DkNN is a step towards correctly handling malicious inputs like adversarial inputs because:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;outputs more reliable confidence estimates on adversarial examples than the softmax.&lt;/li&gt;
&lt;li&gt;provides insights as to why adversarial examples affect undefended DNNs.&lt;/li&gt;
&lt;li&gt;robust to adaptive attacks they considered&lt;/li&gt;
&lt;/ul&gt;

&lt;h2 id=&#34;accuracy-on-adversarial-examples&#34;&gt;Accuracy on Adversarial Examples&lt;/h2&gt;

&lt;p&gt;They crafted adversarial examples using three algorithms: Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), and Carlini-Wagner 2 attack (CW).&lt;/p&gt;

&lt;p&gt;All there test results are shown in the following table. They have also included the accuracy of both undefended DNN and DkNN. By observing the table, they made a conclusion that even though the attacks were successful in evading the undefended DNN, but when the model is integrated with DkNN, then some accuracy on adversarial examples is recovered.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/evaluation-robustness.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Adversarial example classification accuracy for
the DNN and DkNN: attack parameters are chosen according
to prior work. All input features were clipped to remain in their
range. Note that most wrong predictions made by the DkNN
are assigned low credibility (see Figure 6 and the Appendix).    
&lt;/p&gt; 

&lt;p&gt;They have plotted the reliability diagrams comparing the DkNN credibility on GTSRB adversarial examples with the softmax probabilities output by the DNN. For DkNN, credibility is low across all attacks. Here, the number of points in each bin is reflected by the red line. DkNN outputs a credibility below 0.5 for most of the inputs. This indicates a sharp departure from softmax probabilities, which classified most adversarial examples in the wrong class with a high confidence of above 0.9 for the FGSM and BIM attacks. They also made an observation that the BIM attack is more successful at introducing perturbations than that of the FGSM or the CW attacks.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/evaluation_confidence_dknn_adversarial.png&#34; width=&#34;600&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt;  Reliability Diagrams on Adversarial Examples—
The DkNN’s credibility is better calibrated (i.e., it assigns low
confidence to adversarial examples) than probabilities output by
the softmax of an undefended DNN. All diagrams are plotted
with GTSRB test data. Similar graphs for the MNIST and
SVHN datasets are in the Appendix.    
&lt;/p&gt; 

&lt;h2 id=&#34;explanation-of-dnn-mispredictions&#34;&gt;Explanation of DNN Mispredictions&lt;/h2&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/misprediction_dknn.png&#34; width=&#34;600&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Number of Candidate Labels among k = 75 Nearest
Neighboring Representations—Shown for GTSRB with clean
and adversarial data across the layers of the DNN underlying
the DkNN. Points are centered according to the number of
labels found in the neighbors; while the area of points is
proportional to the number of neighbors whose label matches
the DNN prediction. Representations output by lower layers
of the DNN are less ambiguous for clean data than adversarial
examples (nearest neighbors are more homogeneously labeled).    
&lt;/p&gt; 

&lt;p&gt;In the above figure, for both clean and adversarial examples, we can observe that the number of candidate labels decreases as we move up the neural network from its input layer all the way to its output layer. Number of candidate labels (in this case k = 75 nearest neighboring training representations) that match the final prediction made by the DNN is smaller for some attacks. For CW attack, the true label of adversarial examples that it produces is often recovered by the DkNN. Again, the lack of conformity between neighboring training representations at different layers of the model characterizes the weak support for the model’s prediction.&lt;/p&gt;

&lt;!-- ## Robustness of DKNN to Adaptive Attacks

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/robustness_dknn_adaptive.png&#34; width=&#34;600&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt;  Feature Adversarial Examples against our DkNN
algorithm—Shown for SVHN (see Appendix for MNIST).
Adversarial examples are organized according to their original
label (rows) and the DkNN’s prediction (columns).   
&lt;/p&gt;  --&gt;

&lt;!-- ## Conclusion --&gt;

&lt;h3 id=&#34;comparison-to-lid&#34;&gt;Comparison to LID&lt;/h3&gt;

&lt;p&gt;We discussed similarities between the DkNN approach and Local
Intrinsic Dimensionality [&lt;a
href=&#34;https://openreview.net/forum?id=B1gJ1L2aW&#34;&gt;ICLR 2018&lt;/a&gt;]. There
are important differences between the approaches, but given the
results reported in the &lt;em&gt;Obfuscated Gradients Give a False Sense of
Security: Circumventing Defenses to Adversarial Examples&lt;/em&gt; (discussed
in &lt;a href=&#34;https://secml.github.io/class3/&#34;&gt;Class 3&lt;/a&gt;) reported on
LID, it is worth investigating how robust DkNN is to the same
attacks. (Note that neither of these defenses are really obfuscating
gradients, so the attack strategy reported in that paper is to just use high confidence adversarial examples.)&lt;/p&gt;

&lt;h2 id=&#34;the-secret-sharer-measuring-unintended-neural-network-memorization-extracting-secrets&#34;&gt;The Secret Sharer: Measuring Unintended Neural Network Memorization &amp;amp; Extracting Secrets&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Nicholas Carlini, Chang Liu, Jernej Kos, Ulfar Erlingsson, Dawn Song. 2018. The Secret Sharer: Measuring Unintended Neural Network Memorization &amp;amp; Extracting Secrets.  arXiv:1802.08232. &lt;a href=&#34;https://arxiv.org/pdf/1802.08232.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This paper focuses an adversary targeting &amp;ldquo;secret&amp;rdquo; user information stored in a deep neural network. Sensitive or &amp;ldquo;secret&amp;rdquo; user information can be included in the datasets used to train deep machine learning models. For example, if a model is trained on a dataset of emails, some of which have credit card numbers, there is a high probability that the credit card number can be extracted from the model, according to this paper.&lt;/p&gt;

&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;

&lt;p&gt;Rapid adoption of machine learning techniques has resulted in models trained on sensitive user information or &amp;ldquo;secrets&amp;rdquo;. The &amp;ldquo;secrets&amp;rdquo; may include “person messages, location histories, or medical information.” The potential for machine learning models to memorize or store secret information could reveal sensitive user information to an adversary. Even black-box models were found to be susceptible to leaking secret user information. As more deep learning models are implemented, we need to be mindful of the models ability to store information, and to shield models from revealing secrets.&lt;/p&gt;

&lt;h2 id=&#34;contributions&#34;&gt;Contributions&lt;/h2&gt;

&lt;p&gt;The exposure metric defined in this paper measures the ability of a model to memorize a secret. The higher the exposure of a model, the more likely a model is memorizing secret user information. The paper uses this metric to compare the ability of different models to memorize with different hyper-parameters. An important observation was that secrets were found to be memorized early in training rather than in the period of over-fitting. The author found that this property was consistent between different models and hyper parameters. Models included in paper focus on deep learning generative text models. Convolution neural network were also tested, but perform worse generally on text-based data. Extraction of secret information from the model was tested with varying hyper-parameters and conditions.&lt;/p&gt;

&lt;h2 id=&#34;perplexity-and-exposure&#34;&gt;Perplexity and Exposure&lt;/h2&gt;

&lt;p&gt;Perplexity is a measurement of how well a probability distribution predicts a sample. A model will have completely memorized the randomness of the training set if the log-perplexity of a secret is the absolute smallest.&lt;/p&gt;

&lt;p&gt;Log-perplexity suggests memorization but does not yield general information about the extent of memorization in the model. Thus the authors define rank:&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_risk.png&#34; width=&#34;400&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt;  Definition of rank. Requires computing log-perplexity of all possible secrets.
&lt;/p&gt; 

&lt;p&gt;To compute the rank, you must iterate over all possible secrets. To avoid this heavy computation the authors define multiple numerical methods to compute a value related to the rank of a secret, exposure.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_exposure.png&#34; width=&#34;400&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt;  Definition of exposure. More simply, the negative log-rank of a secret, s[r].
&lt;/p&gt; 

&lt;h2 id=&#34;secret-extraction-methods&#34;&gt;Secret Extraction Methods&lt;/h2&gt;

&lt;p&gt;The authors identify four methods for extracting secrets from a black-box model.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Brute Force&lt;/li&gt;
&lt;li&gt;Generative Sampling&lt;/li&gt;
&lt;li&gt;Beam Search&lt;/li&gt;
&lt;li&gt;Shortest Path Search&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Brute force was determined to be too computationally expensive, as the randomness space is too large. Generative sampling and beam search both fail to give the most optimal solution. The author&amp;rsquo;s used shortest path search to guarantee the lowest log-perplexity solution. Their approach was based on Dijkstra&amp;rsquo;s graph algorithm, and is explained in the paper. The figure below demonstrates the process of maximizing the log-perplexity for the purpose of finding the secret.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_dsk.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt;  Shortest path search algorithm. Blue path points to location of largest perplexity, the location of the secret.
&lt;/p&gt; 

&lt;h2 id=&#34;characterizing-memorization-of-secrets&#34;&gt;Characterizing Memorization of Secrets&lt;/h2&gt;

&lt;p&gt;In order to better understand model memorization the authors tested different numbers of iterations, various model architectures, multiple training strategies, changes in secret formats and context, and memorization across multiple simultaneous secrets.&lt;/p&gt;

&lt;h4 id=&#34;iterations&#34;&gt;Iterations&lt;/h4&gt;

&lt;p&gt;As showing in the figure below, the authors collected data relating to the exposure of the model throughout training. The exposure of the secret can clearly be seen rising until the test error starts to rise. This increase in exposure expresses the model&amp;rsquo;s memorization of the training set early in the training process.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_epoch.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt;  Estimated exposure of secret and loss during training.
&lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;model-architectures&#34;&gt;Model architectures&lt;/h4&gt;

&lt;p&gt;The authors observed, through their exposure metric, that memorization was a shared property among recurrent and convolutional neural networks. Data relating to the different model architectures and exposure is shown in the figure below.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_modelComparison.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Table:&lt;/b&gt;  Model Exposure Comparison
&lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;training-strategies&#34;&gt;Training strategies&lt;/h4&gt;

&lt;p&gt;As seen in the figure below, smaller batch sizes resulted in lower levels of memorization. Although having a smaller batch size slows down distributed processing of models, it is clear that their results suggest that a smaller batch size can reduce memorization.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_batchsize.png&#34; width=&#34;400&#34;&gt;
&lt;br&gt; &lt;b&gt;Table:&lt;/b&gt;  Estimated exposure of various model and batch sizes.
&lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;secret-formats-and-context&#34;&gt;Secret formats and context&lt;/h4&gt;

&lt;p&gt;The table below interestingly suggests that the context of secrets significantly impacts whether an adversary can detect memorization. As there become more characters or information associated with the secret, the adversary has an easier time extracting randomness.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_secret.png&#34; width=&#34;400&#34;&gt;
&lt;br&gt; &lt;b&gt;Table:&lt;/b&gt;  Estimated exposure with different contexts.
&lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;memorization-across-multiple-simultaneous-secrets&#34;&gt;Memorization across multiple simultaneous secrets&lt;/h4&gt;

&lt;p&gt;The table below shows the effect of inserting multiple secrets into the dataset. As the number of insertions increase, the model becomes more likely to memorize the secrets that were inserted.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class8/secsha_multiplephrases.png&#34; width=&#34;500&#34;&gt;
&lt;br&gt; &lt;b&gt;Table:&lt;/b&gt; Percentage of phrases that can be extracted from the model
&lt;/p&gt;&lt;/p&gt;

&lt;h2 id=&#34;evaluating-word-level-models&#34;&gt;Evaluating word level models&lt;/h2&gt;

&lt;p&gt;An interesting observation was that the capacity of a large word-level model produced better results than the smaller character-level model. By applying the exposure metrics to a word-level model, the authors found that representation of numbers mattered heavily in the memorization of information. When the authors replaced replaced &amp;ldquo;1&amp;rdquo; with &amp;ldquo;one,&amp;rdquo; they saw the exposure dropped by more than half from 25 to 12 for the large word-level model. Because the word-level model was more than 80 times the size of the character-level model, the authors found it surprising that &amp;ldquo;[the large model] has sufficient capacity to memorize the training data completely, but it actually memorizes less.&amp;rdquo;&lt;/p&gt;

&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;

&lt;p&gt;This paper examines the extent to which memorization has occurred in different types of algorithms and how sensitive user information can be revealed through this memorization. The metrics used in the paper can be easily transferred to existing models that have a well-defined notion of perplexity. They also demonstrate that these metrics can also be used to extract secret user information from black-box models.&lt;/p&gt;

&lt;p&gt;&amp;mdash; Team Panda: Christopher Geier, Faysal Hossain Shezan, Helen Simecek, Lawrence Hook, Nishant Jha&lt;/p&gt;

&lt;h3 id=&#34;sources&#34;&gt;Sources&lt;/h3&gt;

&lt;blockquote&gt;
&lt;p&gt;Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. In Proceedings of ACM Symposium on Operating Systems Principles (SOSP ’17). ACM, New York, NY, USA, 18 pages. https: //doi.org/10.1145/3132747.3132785 &lt;a href=&#34;https://arxiv.org/pdf/1705.06640.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Nicholas Carlini, Chang Liu, Jernej Kos, Ulfar Erlingsson, Dawn Song. 2018. The Secret Sharer: Measuring Unintended Neural Network Memorization &amp;amp; Extracting Secrets.  arXiv:1802.08232. Retrieved from &lt;a href=&#34;https://arxiv.org/pdf/1802.08232.pdf&#34;&gt;https://arxiv.org/pdf/1802.08232.pdf&lt;/a&gt; &lt;a href=&#34;https://arxiv.org/pdf/1802.08232.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Nicolas Papernot, Patrick McDaniel. 2018. Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. Retrieved from &lt;a href=&#34;https://arxiv.org/pdf/1803.04765.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
</description>
    </item>
    
    <item>
      <title>Class 7: Biases in ML, Discriminatory Advertising</title>
      <link>https://secml.github.io/class7/</link>
      <pubDate>Tue, 20 Mar 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class7/</guid>
      <description>

&lt;h2 id=&#34;motivation&#34;&gt;Motivation&lt;/h2&gt;

&lt;p&gt;Machine learning algorithms are playing increasingly important roles in many critical decision making tasks. However, studies reveal that machine learning models are subject to biases, some of which stem from historical biases in human world that are captured in training data. Understanding potential bias, identifying and fixing existing bias can help people design more objective and reliable decision making systems based on machine learning models.&lt;/p&gt;

&lt;h2 id=&#34;ad-transparency&#34;&gt;Ad Transparency&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Athanasios Andreou, Giridhari Venkatadri, Oana Goga, Krishna P. Gummadi, Patrick Loiseau, Alan Mislove. &lt;em&gt;Investigating Ad Transparency Mechanisms in Social Media: A Case Study of Facebook’s Explanations&lt;/em&gt;. NDSS, 2018. &lt;a href=&#34;http://wp.internetsociety.org/ndss/wp-content/uploads/sites/25/2018/02/ndss2018_10-1_Andreou_paper.pdf&#34;&gt;[PDF]&lt;/a&gt; (All images below are taken from this paper.)&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3 id=&#34;what-are-ad-transparency-mechanisms-in-social-media&#34;&gt;What are ad transparency mechanisms in social media？&lt;/h3&gt;

&lt;p&gt;Transparency mechanisms are solutions for many privacy complaints from users and regulators. Users have little understanding of what data the advertising platforms have about them and why they are shown particular ads. The transparency mechanisms provided by Facebook are (1) the &amp;ldquo;Why am I seeing this?&amp;rdquo; button that provides users with an explanation of why they were shown a particular ad (&lt;em&gt;ad explanations&lt;/em&gt;), and (2) an Ad Preferences Page that provides users with a list of attributes Facebook has inferred about them and how (&lt;em&gt;data explanations&lt;/em&gt;). Twitter offers similar transparency mechanisms, including a &amp;ldquo;Why am I seeing this?&amp;rdquo; botton that provides  users with an explanation of why they were shown a particular ad (ad explanations) in Twitter.&lt;/p&gt;

&lt;h3 id=&#34;what-did-this-paper-do&#34;&gt;What did this paper do?&lt;/h3&gt;

&lt;p&gt;This paper reports on an investigation and analysis of ad explanations (why users see a certain ad), and an investigation of data explanations (how data is inferred about a user), which is strongly related to the ad transparency mechanisms in Facebook. This paper first introduced how ad platform in Facebook works and then evaluate the two transparency mechanisms we introduced before with some properties.&lt;/p&gt;

&lt;h3 id=&#34;the-ad-platform-in-facebook&#34;&gt;The Ad platform in Facebook&lt;/h3&gt;

&lt;p&gt;There are three main processes in the Facebook ad platform: a) the data inference process; b) the audience selection process; c) the user-ad matching process.&lt;/p&gt;

&lt;p&gt;(a) The data inference process is the process that allows the advertising platform to learn the users’ attributes. It has three parts: (1) the raw user data (the inputs), containing the information the
advertising platform collects about a user either online or offline; (2) the data inference algorithm (the mapping function between inputs and outputs), covering the algorithm the advertising platform uses to translate input user data to targeting attributes; (3) the resulting targeting attributes (the outputs) of each user that advertisers can specify to select different groups of users.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/a.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/a.png&#34; width=80%&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;(b) The audience selection process is the interface that allows advertisers to express who should receive their ads. Advertisers create audiences by specifying the set of targeting attributes the audience needs to satisfy. Later, to launch an ad campaign, advertisers also need to specify a bid price and an optimization criterion.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/b.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/b.png&#34; width=80%&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;&amp;#40;c) The user-ad matching process takes place whenever someone is eligible to see an ad. It examines all the ad campaigns placed by different advertisers in a particular time interval, their bids, and runs an auction to determine which ads are selected.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/c.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/c.png&#34; width=80%&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;h3 id=&#34;ad-explanations-and-the-experiments-on-this-transparency-mechanism&#34;&gt;Ad Explanations and the experiments on this transparency mechanism&lt;/h3&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/WhySeeingThis.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/WhySeeingThis.png&#34; width=80%&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;As you can see in the picture, there are both attritubutes and
potentional attritubutes here.&lt;/p&gt;

&lt;p&gt;This paper used 5 different properties to evaluate the performance of
Ad explanations:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Correctness: Every attribute listed was used by the advertiser&lt;/li&gt;
&lt;li&gt;Personalization: The attributes listed are unique to the individual&lt;/li&gt;
&lt;li&gt;Completeness: If all relevant attributes are included in the explanation&lt;/li&gt;
&lt;li&gt;Consistency: Users with the same attributes see the same explanations&lt;/li&gt;
&lt;li&gt;Determinism: A user would see the same explanation for ads based on the same target attributes&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The paper evaluated the ad explanations by using Chrome browser extension to record ads and explanations. The experiment had 35 users&amp;rsquo; data across 5 months. This experiment also evaluated the data explanation. This paper made simple statistics for the explanation (see the following figure). And then, it shows the results of different properties on this experiment.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/stat.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/stat.png&#34; width=95%&#34;&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;h3 id=&#34;data-explanations-experiments&#34;&gt;Data Explanations Experiments&lt;/h3&gt;

&lt;p&gt;The data explanations is applied in &amp;ldquo;Your interests&amp;rdquo; part as shown in the following picture.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/like.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/like.png&#34; width=80%&#34;&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;The properties here are not the same as those used in the Ad explanation part. There are 3 new properties.
1. Specificity: A data explanation is precise if it shows the precise activities that were used to infer an attribute about a user.
2. Snapshot completeness: A data explanation is snapshot complete if the explanation shows all the inferred attributes about the user that Facebook makes available.
3. Temporal completeness: a temporally complete explanation is one where the platform shows all inferred attributes over a specified period of time.&lt;/p&gt;

&lt;p&gt;The results of different properties on this experiment are showed below:&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/DataRes.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/DataRes.png&#34; width=&#34;80%&#34;&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;h3 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h3&gt;

&lt;p&gt;While the Ad Preferences Page does bring some transparency to the different attributes users can be targeted with,
the provided explanations are incomplete and often vague. Facebook does not provide information about data broker-provided attributes in its data explanations or in its ad explanations.&lt;/p&gt;

&lt;h2 id=&#34;discrimination-in-online-targeted-advertising&#34;&gt;Discrimination in Online Targeted Advertising&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Till Speicher, Muhammad Ali, Giridhari Venkatadri, Filipe Nunes Ribeiro, George Arvanitakis, Fabr&amp;iacute;cio Benevenuto, Krishna P. Gummadi, Patrick Loiseau, Alan Mislove. &lt;em&gt;Potential for Discrimination in Online Targeted Advertising&lt;/em&gt;. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81:5-19, 2018. &lt;a href=&#34;http://proceedings.mlr.press/v81/speicher18a/speicher18a.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Recently, online targeted advertising platforms like Facebook have received intense criticism for allowing advertisers to discriminate against users belonging to protected groups.&lt;/p&gt;

&lt;p&gt;Facebook, in particular, is facing a civil rights lawsuit for allowing advertisers to target ads using an attribute called &amp;ldquo;ethnic affinity.&amp;rdquo; Facebook has clarified that &amp;ldquo;ethnic affinity&amp;rdquo; does not represent ethnicity, but rather represents a user’s affinity for content related to different ethnic communities. Facebook has agreed to rename the attribute to &amp;ldquo;multicultural affinity&amp;rdquo; and to disallow using this attribute to target ads related to housing, employment, and financial services.&lt;/p&gt;

&lt;p&gt;However, Facebook offers many different ways to describe a set of targeted users, so it’s not adequate to disallow targeting on certain attributes. In this paper, the authors develop a framework for quantifying ad discrimination and show the potential for discriminatory advertising using the three different targeting methods on Facebooks advertising platform: personally identifiable information (PII)-based targeting, attribute-based targeting, and look-alike audience targeting.&lt;/p&gt;

&lt;h3 id=&#34;quantifying-ad-discrimination&#34;&gt;Quantifying Ad Discrimination&lt;/h3&gt;

&lt;p&gt;The authors identify three potential approaches to quantifying discrimination.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Based on advertiser’s intent:&lt;/strong&gt; The authors reject this approach since it is hard to measure and it does not capture unintentionally discriminatory ads.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Based on ad targeting process:&lt;/strong&gt; This category includes existing anti-discrimination measures, like disallowing use of sensitive attributes when defining a target population. The authors reject this approach since it breaks down when there exist several methods of targeting a population.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Based on targeted audience (outcomes):&lt;/strong&gt; This approach takes into account only which users are targeted, not how they are targeted. The authors use this approach to quantify ad discrimination since outcome-based analyses generalize independent of targeting methods.&lt;/p&gt;

&lt;p&gt;The authors formalize outcome-based discrimination as follows:&lt;/p&gt;

&lt;p&gt;&lt;blockquote&gt;
Let \(\mathbf{D} = (u_i)_{i=1,\ldots,n}\) be a database of user records \(u_i\).&lt;br /&gt;
Let \(u_i \in \mathbb{B}^m\) be a vector of \(m\) boolean attributes.&lt;br /&gt;
Let \(s \in {1, \ldots, m}\) be the sensitive attribute we are considering.&lt;br /&gt;
Let \(u_s\) be the value of sensitive attribute \(s\) for user \(u\).&lt;br /&gt;
Let \(\mathbf{S} = {u \in \mathbf{D} | u_s = 1}\) be the set of all users having sensitive attribute \(s\).
   &lt;/blockquote&gt;&lt;/p&gt;

&lt;p&gt;The authors define a metric for how discriminatory an advertiser’s targeting is, inspired by the disparate impact measure used for recruiting candidates from a pool.&lt;/p&gt;

&lt;p&gt;&lt;blockquote&gt;
Let \(\mathbf{TA}\) (target audience) be the set of users selected by the targeting process.&lt;br /&gt;
Let \(\mathbf{RA}\) (relevant audience) be the set of all users in the database \(\mathbf{D}\) who would find the ad useful and interesting.&lt;br /&gt;
   &lt;/blockquote&gt;&lt;/p&gt;

&lt;p&gt;Define the representation ratio measure to capture how much more likely a user is to be targeted when having the sensitive attribute than if the user did not have the attribute:&lt;/p&gt;

&lt;p&gt;$$\text{rep_ratio}_s(\mathbf{TA}, \mathbf{RA}) = \dfrac{|\mathbf{TA} \cap \mathbf{RA}_s|/|\mathbf{RA}_s|}{|\mathbf{TA} \cap \mathbf{RA}_{\neg s}|/|\mathbf{RA}_{\neg s}|}$$&lt;/p&gt;

&lt;p&gt;where \(\mathbf{RA}_s = {u \in \mathbf{RA} | u_s = 1 }\) is the subset of the relevant audience with the sensitive attribute and \(\mathbf{RA}_{\neg s} = {u \in \mathbf{RA} | u_s = 0}\) is the complementary subset of the relevant audience without the sensitive attribute&lt;/p&gt;

&lt;p&gt;Define the disparity in targeting measure to capture both over- and under-representation of a sensitive attribute in a target audience:&lt;/p&gt;

&lt;p&gt;$$\text{disparity}_s(\mathbf{TA}, \mathbf{RA}) = \max\left(\text{rep_ratio}_s(\mathbf{TA}, \mathbf{RA}), \dfrac{1}{\text{rep_ratio}_s(\mathbf{TA}, \mathbf{RA})}\right)$$&lt;/p&gt;

&lt;p&gt;Disparity must be computed based on the relevant audience \(\mathbf{RA}\) because \(\mathbf{RA}\) may have a different distribution of the sensitive attribute than the whole database \(\mathbf{D}\). The authors assume that sensitive attributes considered have the same distributions in the relevant audience as the global population, and therefore high disparity in targeting is evidence of discrimination. Following the &amp;ldquo;80%&amp;rdquo; disparate impact rule, a reasonable disparity threshold for a group to be over- or under-represented may be \(\max(0.8, 1/0.8) = 1.25\).&lt;/p&gt;

&lt;p&gt;The recall of an ad quantifies how many of the relevant users with the sensitive attribute the ad targets or excludes:&lt;/p&gt;

&lt;p&gt;$$\text{recall}(\mathbf{TA}, \mathbf{RA}&amp;lsquo;) = \dfrac{|\mathbf{TA} \cap \mathbf{RA}&amp;lsquo;|}{|\mathbf{RA}&amp;lsquo;|}$$&lt;/p&gt;

&lt;p&gt;where \(\mathbf{RA}&amp;rsquo;\) is one of \(\mathbf{RA}_s\) or \(\mathbf{RA}_{\neg s}\) depending on whether we’re considering the inclusion or exclusion of \(\mathbf{S}\).&lt;/p&gt;

&lt;h3 id=&#34;pii-based-targeting&#34;&gt;PII-Based Targeting&lt;/h3&gt;

&lt;p&gt;PII-based targeting on the Facebook advertising platform allows advertisers to select a target audience using unique identifiers, like phone numbers, email addresses, and combinations of name with other attributes (e.g. birthday or zip code). The authors show that public data sources, such as voter records and criminal history records, contain sufficient PII to construct a discriminatory target audience for a sensitive attribute without explicitly targeting that attribute.&lt;/p&gt;

&lt;p&gt;The authors constructed datasets to show that they could implicitly target gender, race, and age using North Carolina voter records. Each of these attributes is listed in voting records, and the remaining fields together uniquely identify the voter (i.e., last name, first name, city, state, zip code, phone number, and country). The authors uploaded datasets targeting values of each attribute and recorded Facebook’s estimated audience size.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/table1.jpg&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/table1.jpg&#34; width=95%&gt;&lt;/img&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;The Voter Records column shows the distribution of attribute values in the voter records data set. For a given attribute, the Facebook Users column shows how many of the 10,000 people in the dataset constructed for that attribute are actually targetable on Facebook (as reported by the Facebook advertising platform). The final column shows the portion of the targetable users who actually match the targeted attribute, found by restricting the target audience using Facebook’s records of the sensitive attribute. High targetable percentages values show that the voter records overlap significantly with the voter records data set. High validation percentages show that the auxiliary PII was highly accurate at describing particular users with the targeted attribute. Note that there are some low validation percentages, which the authors attribute to Facebook’s inaccurate or incomplete records of some data (for example, they do not know race, only &amp;ldquo;multicultural affinity&amp;rdquo;).&lt;/p&gt;

&lt;h3 id=&#34;attribute-based-targeting&#34;&gt;Attribute-Based Targeting&lt;/h3&gt;

&lt;p&gt;Attribute-based targeting allows advertisers to select a target audience by specifying that targeted users should have some attribute or combination of attributes. The authors group these attributes into two categories: curated attributes and free-form attributes. Curated attributes are well-defined binary attributes spanning demographics, behaviors, and interests — Facebook tracks a list of over 1,100 of these. Free-form attributes describe users inferred interest in entities such as websites and apps as well as topics such as food preferences or niche interests. The authors estimate that there are at least hundreds of thousands of free-form attributes.&lt;/p&gt;

&lt;p&gt;The authors demonstrate that many curated attributes are correlated with sensitive attributes like race, and can therefore be used for discriminatory audience creation. The following table shows experimental results obtained by uploading sets of voter records filtered to contain only a single race and measuring Facebook’s reported size of the subaudiences for each curated attribute. The figures in parentheses are the recall and representation ratio for a population from North Carolina. The top three most inclusive and exclusive attributes per ethnicity are listed. Note the high representation ratios for the &amp;ldquo;Most inclusive&amp;rdquo; column and the low representation ratios for the &amp;ldquo;Most exclusive&amp;rdquo; column.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class7/table2.jpg&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;The authors similarly demonstrated that free-form attributes could used in a discriminatory manner. For example, targeting a vulnerable audience could be made possible by targeting the free-form attributes &amp;ldquo;Addicted,&amp;rdquo; &amp;ldquo;REHAB,&amp;rdquo; &amp;ldquo;AA,&amp;rdquo; or &amp;ldquo;Support group.&amp;rdquo; The authors also showed how Facebook’s attribute suggestions feature could be used to discover new highly-discriminatory free-form attributes. For example, starting a search with &amp;ldquo;Fox&amp;rdquo; (37% conservative audience on Facebook) and following a chain of suggestions leads to &amp;ldquo;The Sean Hannity Show&amp;rdquo; (95% conservative audience on Facebook).&lt;/p&gt;

&lt;h3 id=&#34;look-alike-audience-targeting&#34;&gt;Look-Alike Audience Targeting&lt;/h3&gt;

&lt;p&gt;Look-alike audience targeting allows advertisers to generate a new target audience that looks similar to an existing set of users (the fans of one of their Facebook pages or an uploaded PII data set). The authors show that this feature can be used to scale a biased audience to a much larger population. Experimental results suggest that Facebook attempts to determine the attributes that distinguish the base target audience from the general population and propagates these biases to the look-alike audience. The authors show that this bias propagation can amplify both intentionally created and unintentionally overlooked biases in source audiences.&lt;/p&gt;

&lt;h2 id=&#34;algorithmic-transparency-via-quantitative-input-influence&#34;&gt;Algorithmic Transparency via Quantitative Input Influence&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Anupam Datta, Shayak Sen, Yair Zick. &lt;em&gt;Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems&lt;/em&gt;. 2016 IEEE Symposium on Security and Privacy (&amp;ldquo;Oakland&amp;rdquo;). &lt;a href=&#34;https://www.andrew.cmu.edu/user/danupam/datta-sen-zick-oakland16.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Machine learning systems are increasingly being used to make important societal decisions, in sectors including healthcare, education, and insurance.
For instance, an ML model may help a bank decide if a client is eligible for a loan, and both parties may to know critical details about how the model works.
A rejected client will likely want to know why they were rejected: would they have been accepted if their income was higher?
The answer would be especially important if their reported income was lower than their actual income;
more generally, the client can ensure that their input data contained no errors.&lt;/p&gt;

&lt;p&gt;Conversely, the model&amp;rsquo;s user may want to ensure that the model does not discriminate based on sensitive inputs, such as the legally-restricted features of race and gender.
Simply ignoring those features may not be sufficient to prevent discrimination; e.g., ZIP code can be used as a proxy for race.
This paper proposes a method to solve these problems by making the model&amp;rsquo;s behavior more transparent: a quantitative measure of the effect of a particular feature (or set of features) on the model&amp;rsquo;s decision for an individual.
The paper offers several approaches suited for various circumstances, but they all fall under the umbrella of &amp;ldquo;quantitative input inflence&amp;rdquo;, or QII.&lt;/p&gt;

&lt;h3 id=&#34;unary-qii&#34;&gt;Unary QII&lt;/h3&gt;

&lt;p&gt;The simplest quantitative measure presented is unary QII, which measures the influence of one attribute on a quantity of interest \(Q_\mathcal{A}\) for some subset of the sample space \(X\).
Formally, unary QII is determined as&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/unaryQII.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/unaryQII.png&#34; width=40%&gt;&lt;/img&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;where the first term is the actual expected value of \(Q_\mathcal{A}\) for this subset, and the second term is the expected value if the feature \(i\) were randomized.&lt;/p&gt;

&lt;p&gt;For example, consider the rejected bank client from above.
If they restrict \(X\) to only contain their feature vector, and they set \(Q_\mathcal{A}\) to output the model&amp;rsquo;s probability of rejection,
then unary QII tells how much any individual feature impacted his loan application.
If the unary QII for a feature is large, changing the value of that feature would likely increase their odds of being accepted;
conversely, changing the value of a feature with low unary QII would make little difference.&lt;/p&gt;

&lt;p&gt;The paper presents a concrete example: Mr. X has been classified as a low-income individual, and he would like to know why.
Since only 2.1% of people with income above $10k are classified as low-income, Mr. X suspects racial bias.
In actuality, the transparency report shows that neither his race nor country of origin were significant;
rather, his marital status and education were far more influential in his classification.&lt;/p&gt;

&lt;p&gt;&lt;center&gt;
&lt;a href=&#34;https://secml.github.io/images/class7/mrxprofile.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/mrxprofile.png&#34; width=&#34;40%&#34;&gt;&lt;/a&gt;&lt;Br&gt;
&lt;A href=&#34;https://secml.github.io/images/class7/mrxreport.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/mrxreport.png&#34; width=50%&gt;&lt;/a&gt;
&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;The sample space \(X\) can also be broadened to include an entire class of people.
For instance, suppose \(X\) is restricted to include people of just one gender, and \(Q_\mathcal{A}\) is set to output the model&amp;rsquo;s probability of acceptance.
Here, unary QII would reveal the influence of a feature \(i\) on men and on women.
A disparity between the two measures may then indicate that the model is biased:
specifically, the feature \(i\) can be identified as a proxy variable, used by the model to distinguish between men and women (even if gender is omitted as an input feature).&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/unarygraph.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/unarygraph.png&#34; width=50%&gt;&lt;/img&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;However, unary QII is often insufficent to explain a model&amp;rsquo;s behavior on an individual or class of individuals.
This histogram shows the paper&amp;rsquo;s results for their &amp;ldquo;adult&amp;rdquo; dataset:
for each individual, the feature that created the highest unary QII was found, and the unary QII value was plotted in the histogram.
Most individuals could not be explained by any particular feature, and most features had little influence by themselves.&lt;/p&gt;

&lt;h3 id=&#34;set-and-marginal-qii&#34;&gt;Set and Marginal QII&lt;/h3&gt;

&lt;p&gt;Thankfully, unary QII can easily be generalized to incorporate multiple features at once.
Set QII is defined as
&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/setQII.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/setQII.png&#34; width=35%&gt;&lt;/img&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;where \(S\) is a set of features (as opposed to a single feature, like \(i\) in unary QII).
The paper also defines marginal QII
&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/marginalQII.png&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/marginalQII.png&#34; width=55%&gt;&lt;/img&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;which measures the influence of a feature \(i\) after controlling for the features in \(S\).
These two quantitative measures have different use cases, but both are more general (and thus more useful) than unary QII.&lt;/p&gt;

&lt;p&gt;Marginal QII can measure the influence of a single feature \(i\), like unary QII, but only for a specific choice of \(S\),
and the amount of influence can vary wildly depending on the choice of \(S\).
To account for this, the paper defines the &lt;em&gt;aggregate influence&lt;/em&gt; of \(i\), which measures the expected influence of \(i\) for random choices of \(S\).&lt;/p&gt;

&lt;h3 id=&#34;conclusion-1&#34;&gt;Conclusion&lt;/h3&gt;

&lt;p&gt;The above variants of QII can be used to provide transparency reports, offering insight into how an ML model makes decisions about an individual.
Malicious actors may seek to abuse such a system, carefully crafting their input vector to glean someone else&amp;rsquo;s private information.
However, these QII measures are shown too have low sensitivity, so differential privacy can be added with small amounts of noise.&lt;/p&gt;

&lt;p&gt;These QII measures are useful only if the input features have well-defined semantics.
This is not true in domains such as image or speech recognition, yet transparency is still desirable there.
The authors assert that designing transparency mechanisms in these domains is an important future goal.
Nevertheless, these QII measures are remarkably effective on real datasets, both for understanding individual outcomes and for finding biases in ML models.&lt;/p&gt;

&lt;h2 id=&#34;language-corpus-bias&#34;&gt;Language Corpus Bias&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Caliskan, A., Bryson, J., &amp;amp; Narayanan, A. (2017). &lt;em&gt;Semantics derived automatically from language corpora contain human-like biases&lt;/em&gt;. Science, 356(6334), 183-186. doi:10.1126/science.aal4230 &lt;a href=&#34;http://science.sciencemag.org/content/sci/356/6334/183.full.pdf&#34;&gt;[PDF]&lt;/a&gt; [&lt;a href=&#34;http://opus.bath.ac.uk/55288/4/CaliskanEtAl_authors_full.pdf&#34;&gt;Author&amp;rsquo;s Full Version PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The focus of this paper is how machine learning can learn from the biases and stereotypes in humans. The main contributions of the authors are:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Using word embeddings to extract associations in text&lt;/li&gt;
&lt;li&gt;Replicate human bias to reveal prejudice behavior in humans&lt;/li&gt;
&lt;li&gt;Show that cultural stereotypes propagate to widely used AI today&lt;/li&gt;
&lt;/ol&gt;

&lt;h3 id=&#34;uncovering-biases-in-ml&#34;&gt;Uncovering Biases in ML&lt;/h3&gt;

&lt;p&gt;The authors began by replicating inoffensive biases using their original Word-Embedding Association Test (WEAT) method. Word embedding is a representation of words in vector space. WEAT is a test applied to words in AI which represents words as a 300 dimensional vector. The words are then paired by distances between the vectors. Using WEAT, they demonstrated that flowers have pleasant associations and insects have unpleasant associations. Or instruments are more pleasant than weapons. The word embeddings know the properties of flowers or weapons even though they have no experience with them!&lt;/p&gt;

&lt;p&gt;After showing that WEAT works, they use this technique to show that machine learning absorbs stereotype biases. In a study by Bertrand and Mullainathan, 5,000 identical resumes were sent out to 1,300 job ads and varied only the names. The European American names were 50% more likely to be offered an opportunity to be interviewed. Based on this study, the authors used WEAT to test the pleasantness associations with the names from Bertrand’s work and found European American names were more pleasant than African American names.&lt;/p&gt;

&lt;p&gt;They then turned to studying gender biases. Female names were associated with family as oppose to male names which were associated with career. They also showed woman/girl associated more with arts than math compared to men. These observations were then correlated with data in the labor force. This is show in the figure below:&lt;/p&gt;

&lt;p&gt;&lt;center&gt;&lt;a href=&#34;https://secml.github.io/images/class7/gender_bias.PNG&#34;&gt;&lt;img src=&#34;https://secml.github.io/images/class7/gender_bias.PNG&#34; width=50%&gt;&lt;/img&gt;&lt;/a&gt;&lt;/center&gt;&lt;/p&gt;

&lt;p&gt;The authors then applied another method of their creation called Word-Embedding Factual
Association Test (WEFAT) to show that these embeddings correlate strongly with the occupations women have in the real world. They then used the GloVe method to find similarity between a pair of vectors. Similarity between vectors is related to the probability that the words co-occur with other words similar to each other. GloVe finds this by doing dimensionality reduction to amplify signal in co-occurring probabilities.&lt;/p&gt;

&lt;p&gt;Afterwards, they did a crawl of the internet and got 840 billion words, and each word had a 300 dimension vector derived from counts of other words that occur with it in a 10 word window. WEFAT allowed them to further examine how embeddings capture empirical information. Using this, they were able to predict properties from the given vector.&lt;/p&gt;

&lt;h3 id=&#34;conclusion-2&#34;&gt;Conclusion&lt;/h3&gt;

&lt;p&gt;So what does their work mean? Their results show there’s a way to reveal unknown implicit associations. They demonstrate that word embeddings encode not only stereotyped bias, but also other knowledge like that flowers are pleasant. These results also explain origins of prejudice in humans. It shows how group identity transmits through language before an institution explains why individuals make prejudiced decisions. There are implications for AI and ML because technology could be perpetuating cultural stereotypes. What if ML responsible for reviewing resumes absorbed cultural stereotypes? It’s important to keep this in mind and be cautious in the future.&lt;/p&gt;

&lt;h2 id=&#34;men-also-like-shopping-reducing-bias-amplification&#34;&gt;Men Also Like Shopping: Reducing Bias Amplification&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. &lt;em&gt;Men Also Like Shopping:Reducing Gender Bias Amplification using Corpus-level Constraints&lt;/em&gt;. 2017 Conference on Empirical Methods in Natural Language Processing. &lt;a href=&#34;https://arxiv.org/pdf/1707.09457.pdf&#34;&gt;arXiv preprint arXiv:1709.10207&lt;/a&gt;. July 2017.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Language is increasingly being used to identify some rich visual recognition tasks. And structured prediction models are widely applied to these tasks to take advantage of correlations between co-ocurring labels and visual inputs. However, inadvertently, there can be social biases encoded in the model training procedure, which may magnify some stereotypes and poses challenge in the fairness of (machine learning) model decision making.&lt;/p&gt;

&lt;p&gt;Researchers found that datasets for these tasks contain significant gender bias and models trained on these biased dataset further amplifies these existing biases. An example provided in the paper is, the activity &amp;ldquo;cooking&amp;rdquo; is over 33% more likely to refer females than males in the training set, and a model trained on this dataset can further amplify the disparity of gender ratio to 68% at test time. And to tackle the problem, the author proposed to adopt corpus-level constraints for calibrating existing structured prediction models. Specifically, the author limit the gender bias of the model deviate by only a small amount from what is in the original training data.&lt;/p&gt;

&lt;h3 id=&#34;problem-formulation&#34;&gt;Problem Formulation&lt;/h3&gt;

&lt;p&gt;The problem is then defined as maximizing the test time inference likelihood while also satisfying the corpus-level constraint. A bias score for an output \(o\) with respect to demographic variable \(g\)is defined as:
$$b(o,g) = \frac{c(o,g)}{\sum_{g^{&amp;lsquo;}\in G}c(o,g^{&amp;lsquo;})}$$
where \(c(o,g)\) captures the number of occurrences of \(o\) and \(g\) in a corpus. And a bias might be exhibited if \(b(o,g)&amp;gt;1/||G||\). A mean bias amplification of a model compared to the bias on training data set (i.e., \(b^{*}(o,g)\)) is defined as:&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/mean_bias_amp.png&#34; width=&#34;350&#34; &gt;
&lt;br&gt;
&lt;/p&gt;

&lt;p&gt;with these terms defined, the author proposes the calibration algorithm: \(\textbf{R}educing~\textbf{B}ias~\textbf{A}mplification\) (RBA). Intuitive understanding the of calibration algorithm is to inject constraints to ensure the model predictions follow the gender distribution observed from the training data with allowbable small deviations.&lt;/p&gt;

&lt;h4 id=&#34;structured-output-prediction&#34;&gt;Structured Output Prediction&lt;/h4&gt;

&lt;p&gt;Given a test insatnce, the inference problem at test time is defined as:
$$\underset{y\in Y}{\operatorname{argmax}}f_{\theta}(y,i)$$&lt;/p&gt;

&lt;p&gt;with \(f_{\theta} (y,i)\) is a scoring function based on model \(\theta\) learned from training data. The inference problem hence can be interpreted as finding the structured oupt \(y\) such that the scoring function is maximized. The corpus level constraint is expressed as:&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/corpus_constraint.png&#34; width=&#34;450&#34; &gt;
&lt;br&gt;
&lt;/p&gt;

&lt;p&gt;With the given constraint, the problem is formulated as:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/final_form.png&#34; width=&#34;300&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;where \(i\) refers to insatnce \(i\) in test dataset.&lt;/p&gt;

&lt;p&gt;The corpus level constraint is represented by \(A\sum_{i}y^{i}-b \leq 0\), where the matrix \(A\in R^{l \times K}\) is the coefficients of one constraint, and \(b \in R^{l}\). Note that, above formulation can be solved individually for each instance \(i\).&lt;/p&gt;

&lt;h4 id=&#34;lagrangian-relaxation&#34;&gt;Lagrangian Relaxation&lt;/h4&gt;

&lt;p&gt;The final optimization problem is a mixed integer programming problem, and solving with off-the-shell solver is inefficient for large-scale dataset. Hence, the author proposed to solve the problem with &lt;a href=&#34;https://www.jair.org/media/3680/live-3680-6584-jair.pdf&#34;&gt;Lagrangian relaxation technique&lt;/a&gt;. With a lagrangian multiplier introduced, we have the Lagrangian as
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/lagrangian_form.png&#34; width=&#34;350&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;where \(\lambda_{j} \geq 0, \forall \in {1,&amp;hellip;,l}\). The solution to the problem is then obtained by iteratively optimizing the problem with respect to \(y^{i}\) and \(\lambda\). Specifically, we need two steps in each optimization iteration:&lt;br /&gt;
1) At iteration \(t\), first get the output solution of each instance \(i\)
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/lagrangian_opt_1.png&#34; width=&#34;300&#34; &gt;
&lt;br&gt;
&lt;/p&gt;
&lt;br /&gt;
2) next update the Lagrangian multipliers
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/lagrangian_opt_2.png&#34; width=&#34;300&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;h3 id=&#34;experimental-setup&#34;&gt;Experimental Setup&lt;/h3&gt;

&lt;p&gt;This problem is evvaluated on two vision recognition tasks: visual semantic role labeling (vSRL), and multi-label classification (MLC). The authors focus on the gender bias problem, where \(G = \{man, woman\}\) and focus on the agent and any occurrence in text associated with the images in MLC.&lt;/p&gt;

&lt;h4 id=&#34;dataset-and-model&#34;&gt;Dataset and Model&lt;/h4&gt;

&lt;p&gt;The experiment of vSRL is conducted on &lt;a href=&#34;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Yatskar_Situation_Recognition_Visual_CVPR_2016_paper.pdf&#34;&gt;imSitu&lt;/a&gt; where activity classes are drawn from verbs and roles in &lt;a href=&#34;http://delivery.acm.org/10.1145/990000/980860/p86-baker.pdf?ip=128.143.69.35&amp;amp;id=980860&amp;amp;acc=OPEN&amp;amp;key=B33240AC40EC9E30%2E95F2ACB8D94EAE2C%2E4D4702B0C3E38B35%2E6D218144511F3437&amp;amp;__acm__=1521592411_637f2c8599c29ab1aa3d8bf2818f6140&#34;&gt;FrameNet&lt;/a&gt; and noun categories are drawn from &lt;a href=&#34;https://academic.oup.com/ijl/article-abstract/3/4/235/923280&#34;&gt;WordNet&lt;/a&gt;. The model is  built on the baseline &lt;a href=&#34;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Yatskar_Situation_Recognition_Visual_CVPR_2016_paper.pdf&#34;&gt;CRF&lt;/a&gt; released with the data, which has been shown effective compared to a non-structured prediction baseline [2]. The experiment of MLC is conducted on &lt;a href=&#34;https://link.springer.com/chapter/10.1007/978-3-319-10602-1_48&#34;&gt;MS-COCO&lt;/a&gt;. The model is a similar model as CRF that is used for vSRL.&lt;/p&gt;

&lt;h4 id=&#34;result-analysis&#34;&gt;Result analysis&lt;/h4&gt;

&lt;p&gt;For both the vSRL and MLC tasks, their training data is biased as illustrated in Figure 2. Y-axis denotes the percentage of male agents and x-axis represents gender bias in training data. It is clear that many verbs are biased in the training set and when a model is trained on these biased training datasets, the gender bias is further amplified. Figure 2(a) denotes demonstrates the gender bias in vSRL task and figure 2(b) shows the gender bias in MLC task. Some seemingly neutral words like &amp;ldquo;microwaving&amp;rdquo; and &amp;ldquo;washing&amp;rdquo; is heavily biased towards female and other words like &amp;ldquo;driving&amp;rdquo; is beavily biased towards male.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/bias_fig.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;
&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class7/bias_fig.png&#34; alt=&#34;&#34; /&gt;
&lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://arxiv.org/pdf/1608.04644.pdf)&#34;&gt;&lt;em&gt;Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints&lt;/em&gt;&lt;/a&gt; [1]
&lt;/div&gt;&lt;/p&gt;

&lt;p&gt;Calibration results are then summarized in the table below, which utilizes RBA method. The experimental results show that, with this calibrated method, we are able to significantly reduce the gender bias.&lt;br /&gt;
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class7/calib_result.png&#34; width=&#34;30%&#34;&gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class7/bias_fig.png&#34; alt=&#34;&#34; /&gt;
&lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://arxiv.org/pdf/1707.09457.pdf)&#34;&gt;&lt;em&gt;Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints&lt;/em&gt;&lt;/a&gt; [1]
&lt;/div&gt;&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;—&amp;ndash; Team Gibbon:
Austin Chen, Jin Ding, Ethan Lowman, Aditi Narvekar, Suya&lt;/p&gt;

&lt;h4 id=&#34;references&#34;&gt;References&lt;/h4&gt;

&lt;p&gt;&lt;a href=&#34;http://proceedings.mlr.press/v81/speicher18a/speicher18a.pdf&#34;&gt;[1]&lt;/a&gt; Till Speicher, Muhammad Ali, Giridhari Venkatadri, Filipe Nunes Ribeiro, George Arvanitakis, Fabr&amp;iacute;cio Benevenuto, Krishna P. Gummadi, Patrick Loiseau, Alan Mislove. &amp;ldquo;Potential for Discrimination in Online Targeted Advertising.&amp;rdquo; Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81:5-19, 2018.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.andrew.cmu.edu/user/danupam/datta-sen-zick-oakland16.pdf&#34;&gt;[2]&lt;/a&gt; Anupam Datta, Shayak Sen, Yair Zick. &amp;ldquo;Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems.&amp;rdquo; 2016 IEEE Symposium on Security and Privacy (SP), 2016.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;http://science.sciencemag.org/content/sci/356/6334/183.full.pdf&#34;&gt;[3]&lt;/a&gt; Aylin Caliskan, Joanna J. Bryson, Arvind Narayanan. &amp;ldquo;Semantics derived automatically from language corpora contain human-like biases.&amp;rdquo; Science Magazine, 2017.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.jair.org/media/3680/live-3680-6584-jair.pdf&#34;&gt;[4]&lt;/a&gt; Rush, Alexander M., and Michael Collins. &amp;ldquo;A tutorial on dual decomposition and Lagrangian relaxation for inference in natural language processing.&amp;rdquo; Journal of Artificial Intelligence Research (2012).&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Yatskar_Situation_Recognition_Visual_CVPR_2016_paper.pdf&#34;&gt;[5]&lt;/a&gt; Yatskar, Mark, Luke Zettlemoyer, and Ali Farhadi. &amp;ldquo;Situation recognition: Visual semantic role labeling for image understanding.&amp;rdquo; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;http://delivery.acm.org/10.1145/990000/980860/p86-baker.pdf?ip=128.143.69.35&amp;amp;id=980860&amp;amp;acc=OPEN&amp;amp;key=B33240AC40EC9E30%2E95F2ACB8D94EAE2C%2E4D4702B0C3E38B35%2E6D218144511F3437&amp;amp;__acm__=1521592411_637f2c8599c29ab1aa3d8bf2818f6140&#34;&gt;[6]&lt;/a&gt; Baker, Collin F., Charles J. Fillmore, and John B. Lowe. &amp;ldquo;The berkeley framenet project.&amp;rdquo; Proceedings of the 17th international conference on Computational linguistics-Volume 1. Association for Computational Linguistics, 1998.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://academic.oup.com/ijl/article-abstract/3/4/235/923280&#34;&gt;[7]&lt;/a&gt; Miller, George A., et al. &amp;ldquo;Introduction to WordNet: An on-line lexical database.&amp;rdquo; International journal of lexicography 3.4 (1990): 235-244.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://link.springer.com/chapter/10.1007/978-3-319-10602-1_48&#34;&gt;[8]&lt;/a&gt; Lin, Tsung-Yi, et al. &amp;ldquo;Microsoft coco: Common objects in context.&amp;rdquo; European conference on computer vision. Springer, Cham, 2014.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;http://wp.internetsociety.org/ndss/wp-content/uploads/sites/25/2018/02/ndss2018_10-1_Andreou_paper.pdf&#34;&gt;[9]&lt;/a&gt; Andreou, Athanasios, et al. &amp;ldquo;Investigating ad transparency mechanisms in social media: A case study of Facebook’s explanations.&amp;rdquo; NDSS, 2018.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Class 6: Measuring Robustness of ML Models</title>
      <link>https://secml.github.io/class6/</link>
      <pubDate>Fri, 02 Mar 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class6/</guid>
      <description>

&lt;h2 id=&#34;motivation&#34;&gt;Motivation&lt;/h2&gt;

&lt;p&gt;In what seems to be an endless back-and-forth between new adversarial attacks and new defenses against those attacks, we would like a means of formally verifying the robustness of machine learning algorithms to adversarial attacks.
 In the privacy domain, there is the idea of a differential privacy budget, which quantifies privacy over all possible attacks. In the following three papers, we see attempts at deriving an equivalent benchmark for security, one that will allow the evaluation of defenses against all possible attacks instead of just a specific one.&lt;/p&gt;

&lt;h2 id=&#34;provably-minimally-distorted-adversarial-examples&#34;&gt;Provably Minimally Distorted Adversarial Examples&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Nicholas Carlini, Guy Katz, Clark Barrett, David L. Dill. &lt;em&gt;Provably Minimally-Distorted Adversarial Examples&lt;/em&gt;. &lt;a href=&#34;https://arxiv.org/abs/1709.10207&#34;&gt;arXiv preprint arXiv:1709.10207&lt;/a&gt;. September 2017.&lt;/p&gt;

&lt;p&gt;Guy Katz, Clark Barrett, David Dill, Kyle Julian, Mykel Kochenderfer. &lt;em&gt;Reluplex: An efficient SMT solver for verifying deep neural networks&lt;/em&gt;. International Conference on Computer Aided Verification. 2017. &lt;a href=&#34;https://arxiv.org/pdf/1702.01135.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;There have been many attacking techniques against deep learning models that can effectively generate adversarial examples, such as FGSM, JSMA, DeepFool and the Carlini &amp;amp; Wagner attacks. But most of them couldn&amp;rsquo;t verify the absence of adversarial examples even if they fail to find any result.&lt;/p&gt;

&lt;p&gt;Researchers have started to borrow the idea and techniques from the program verification field to verify the robustness of neural networks. In order to verify the correctness of a program, we can encode a program into SAT-modulo-theories (SMT) formulas and use some off-the-shelf solvers (e.g. Microsoft Z3) to verify a correctness property. An SMT solver generates sound and complete results, either telling you the program never violates the property, or giving you some specific counter-examples. However, we may not be able to get the results for some large programs in our lifetime, because it is an NP-complete problem.&lt;/p&gt;

&lt;p&gt;Similarly, the current neural network verification techniques haven&amp;rsquo;t been able to deal with deep learning models of arbitrary size. But one of the prototype named Reluplex has produced some promising results on the MNIST dataset.&lt;/p&gt;

&lt;p&gt;The Reluplex is an extension of the Simplex algorithm. It introduces a new domain theory solver to take care of the ReLU activation function because the Simplex only deals with linear real arithmetic. You can find more details about Reluplex in:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Guy Katz, Clark Barrett, David Dill, Kyle Julian, Mykel Kochenderfer. &lt;em&gt;Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks&lt;/em&gt;. &lt;a href=&#34;https://arxiv.org/abs/1702.01135&#34;&gt;Arxiv&lt;/a&gt;. 2017.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The paper we discuss here uses Reluplex to verify the &lt;em&gt;local adversarial robustness&lt;/em&gt; of two MNIST classification models. We say a model is delta-locally-robust at input \(x\) if for every \(x&amp;rsquo;\) such that \( ||x-x&amp;rsquo;||p \le \delta \), the model predicts the same label for \(x\) and \(x&amp;rsquo;\). Local robustness is certified for individual inputs, which is substantially different from the &lt;em&gt;global robustness&lt;/em&gt; that certifies the whole input space. The paper performs binary search on delta to find the minimal distortion at a certain precision, and each delta corresponds to one execution instance of Reluplex. The paper only considers the \(L_{infinity}\) norm and the \(L_1\) norm because it is easier to encode these constraints with Reluplex. For example, the \(L_1\) norm could be encoded as
\( |x| = max(x, -x)=ReLu(2x)-x \).&lt;/p&gt;

&lt;p&gt;The evaluation is conducted on a fully-connected, 3-layer network that has 20k weights and fewer than 100 hidden neurons. The testing accuracy of the model is 97%. The model is smaller than most of the state-of-the-art models and has inferior accuracy, but should be good enough for the model verification prototype. The authors arbitrarily selected 10 source images with known labels from the MNIST test set, which produced 90 targeted attack instances in total.&lt;/p&gt;

&lt;p&gt;Even though the Reluplex method is faster than most of the existing general SMT solvers, it is not fast enough for verifying the MNIST classification models. For every configuration with different target models and different \(L_p\) norm constraints, Reluplex always timed out for some of the 90 instances. The experiments with the \(L_1\) constraint were generally slower than those with the
L_infinity constraint, because it introduced more ReLU components. However, we still found some interesting results from the successful verification instances.&lt;/p&gt;

&lt;p&gt;The paper compared the minimally-distorted adversarial examples found by Reluplex with those generated by the CW attack and concluded that iterative optimization-based attacks are effective because the CW attack produced adversarial examples within 12% of optimal on the specific models.&lt;/p&gt;

&lt;p&gt;The paper also evaluated a particular defense technique proposed by Madry et al. which is an adversarial training method that uses the PGD attack and enlarges the model capacity. The paper concluded that the PGD-based adversarial training increased the robustness to adversarial examples by 4.2x on the examined samples. Even though this result doesn&amp;rsquo;t guarantee the efficacy at larger scale, it proves that the defense increases the robustness against all future attacks while it is only trained with the PGD attack.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;evaluating-the-robustness-of-neural-networks&#34;&gt;Evaluating the Robustness of Neural Networks&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, Luca Daniel. &lt;em&gt;Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach&lt;/em&gt;. ICLR 2018. January 2018 &lt;a href=&#34;https://arxiv.org/pdf/1801.10578.pdf&#34;&gt;[PDF]&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Little work has been done towards developing a comprehensive measure of robustness for neural networks, primarily due to them growing mathematically complex as the number of layers increases. The authors contribute a lower bound on the minimal perturbation needed to generate an adversarial sample. They accomplish this by using the extreme value theorem to estimate the local Lipschitz constant for a given sample.&lt;/p&gt;

&lt;p&gt;The work is motivated by the success of a particular attack by Carlini and Wagner against several previous defense strategies such as defensive distillation, adversarial training, and model ensemble &lt;a href=&#34;https://arxiv.org/pdf/1608.04644.pdf&#34;&gt;[2]&lt;/a&gt;. This highlights the need for a means for evaluating a defenses effectiveness against &lt;em&gt;all&lt;/em&gt; attacks rather than just the ones tested against.&lt;/p&gt;

&lt;p&gt;Previous attempts at deriving such a lower bound have shortcomings of their own. Szegedy et al. compute the product of the global Lipschitz constant of each layer of the network to derive a metric of instability of a deep neural network&amp;rsquo;s output; however the global Lipschitz constant is a loose bound &lt;a href=&#34;https://arxiv.org/pdf/1312.6199.pdf&#34;&gt;[3]&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Hein and Andriushchenko derived a closed-form bound using the local Lipschitz constant, but such a bound is only feasible for networks with one hidden layer. Several other approaches used linear programming to verify properties of neural networks, but they are also infeasible for large networks &lt;a href=&#34;https://arxiv.org/pdf/1705.08475.pdf&#34;&gt;[4]&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The most similar work uses a linear programming formulation to derive an &lt;em&gt;upper&lt;/em&gt; bound on the minimum distortion needed, which is not as useful as a universal robustness metric, and also is infeasible for large networks due to the computational complexity of linear programming.&lt;/p&gt;

&lt;h3 id=&#34;robustness-metric&#34;&gt;Robustness metric&lt;/h3&gt;

&lt;p&gt;In order to derive a formal robustness guarantee, the authors formulate the lower bound for the minimum adversarial distortion needed to be a successful perturbed example (success here meaning fooling the classifier, thus becoming adversarial). They accomplish this by relating p-norm of the perturbation \(\lVert \delta \rVert_p \) to the local Lipschitz constant \( L_q^j \), local meaning defined over a l-ball around the input
They use properties of Lipschitz continuous functions to prove the following:&lt;/p&gt;

&lt;p&gt;$$ \lVert \delta \rVert \le \min_{j \ne c}  \frac{f_c(x_0) - f_j(x_0)}{L_q^j} $$&lt;/p&gt;

&lt;p&gt;That is to say that if the the p-norm of the perturbation is less than the difference between any two classifications of the input, \((x_0\)), divided by the local Lipschitz constant.&lt;/p&gt;

&lt;p&gt;The authors go on to provide a similar guarantee for networks where the activation function is not differentiable, for example an ReLU network. In such a case, the local Lipschitz constant can be replaced with the supremum of the directional derivatives for each direction heading towards the non-differentiable point.&lt;/p&gt;

&lt;h3 id=&#34;clever-robustness-metric&#34;&gt;CLEVER Robustness Metric&lt;/h3&gt;

&lt;p&gt;Since the above formulations of the lower bound are difficult to compute, the authors present a technique for estimating the lower bound, which is much more feasible computationally. They make use of the extreme value theorem, which claims that for any random variable, the maximum value of infinite samples of it follows a known distribution. In this case, the random variable is the p-norm of the gradient of a given sample. The authors assume a Weibull non-degenerate distribution for this paper and verify it as a reasonable assumption empirically.&lt;/p&gt;

&lt;p&gt;To apply the theorem, they generate N samples in a ball around a given sample and calculate the gradient norm of each sample. Using the maximum value of the gradient over those N samples, they apply maximum likelihood to get distribution parameters that maximize the probability of those gradients. It should be noted that this approach assumes that the adversarial examples are well-distributed enough that enough random noise will generate them.&lt;/p&gt;

&lt;p&gt;The resulting CLEVER score is an approximate lower bound on the distortion needed for an attack to succeed.&lt;/p&gt;

&lt;h3 id=&#34;experiments-and-results&#34;&gt;Experiments and Results&lt;/h3&gt;

&lt;p&gt;To evaluate how effective an indicator of robustness the CLEVER score is, the authors conducted experiments using the CIFAR-10, MNIST, and ImageNet data sets, pairing each data set with a set of neural network architectures and corresponding popular defenses for those networks.&lt;/p&gt;

&lt;p&gt;They estimate the Weibull distribution parameter and conduct a goodness-of-fit test to verify that the distribution fits the data empirically.&lt;/p&gt;

&lt;p&gt;They then apply two recent state-of-the-art attacks to their models, the Carlini and Wagner attack &lt;a href=&#34;https://arxiv.org/pdf/1608.04644.pdf&#34;&gt;[2]&lt;/a&gt;, covered in a previous blog post, and I-FGSM &lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;[3]&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;To evaluate the CLEVER score&amp;rsquo;s effectiveness, the authors compare it with the average \( L_2 \) and
L_infinity distortions for each adversarial example generated by each type of attack. Since the score is an estimate of the lower bound of the minimal distortion needed, if it is an accurate estimate, then no attack should succeed with a distortion less than the lower bound. Their results show that this holds under both distortion metrics.&lt;/p&gt;

&lt;p&gt;These results also show that the Carlini Wagner attack produces adversarial examples with distortion much closer to minimal than I-FGSM, which demonstrates the CLEVER score&amp;rsquo;s ability to evaluate the strength of attacks themselves. The score also serves as a metric for the effectiveness of of defenses against adversarial attacks since the score was shown to increase for defensively distilled networks. The authors generally contribute a means of providing theoretical guarantees about neural networks that is no limited by the size of the network.&lt;/p&gt;

&lt;p&gt;However, there does not seem to be a correlation between the CLEVER score and the distortion needed by the Carlini Wagner attack. If the score were to truly indicate how hard it is to generate adversarial examples, then we would expect that networks with a higher CLEVER score to have higher average distortions than networks with lower scores, but this was not always the case.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;lower-bounds-on-the-robustness-to-adversarial-perturbations&#34;&gt;Lower bounds on the Robustness to Adversarial Perturbations&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Jonathan Peck, Joris Roels, Bart Goossens, Yvan Saeys &lt;a href=&#34;https://papers.nips.cc/paper/6682-lower-bounds-on-the-robustness-to-adversarial-perturbations.pdf&#34;&gt;Lower bounds on the robustness to adversarial perturbations&lt;/a&gt;. NIPS 2017.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In this paper, the authors propose theoretical lower bounds on the adversarial
perturbations on different types of layers in neural networks. They then combine
these theoretical lower bounds derived layer-by-layer, and apply it to the
entire network to calculate the theoretical lower bound for adversarial
perturbations for the network. In contrast to previous work on this topic, the
authors derive the lower bounds directly in terms of the model parameters.
This is useful for applying the bounds on real-world DNN models.&lt;/p&gt;

&lt;h3 id=&#34;lower-bounds-on-different-classifiers&#34;&gt;Lower bounds on different classifiers&lt;/h3&gt;

&lt;p&gt;The authors use a modular approach to find the lower bound of a feedforward
neural network. In this approach, they derive the lower bound at a particular
layer by working their way backward starting from the output layer, and
gradually towards the input layer. More precisely, given a layer that takes as
input \(y\) and computes a function \(h\) on the input, if we know the
robustness bound of the following layer \(k\), then the goal at the current
layer is to find the perturbation \(r\) such that,&lt;/p&gt;

&lt;p&gt;$$||h(y+r)|| = ||h(y)|| + k$$&lt;/p&gt;

&lt;p&gt;Any adversarial perturbation to that layer, \(q\) is guaranteed to satisfy
\(||q|| \geq ||r||\).&lt;/p&gt;

&lt;p&gt;The lower bounds for different types of layers, as proposed in the paper, is
shown below. The proofs can be found in the &lt;a href=&#34;https://papers.nips.cc/paper/6682-lower-bounds-on-the-robustness-to-adversarial-perturbations-supplemental.zip&#34;&gt;supplemental materials provided by
the authors of the paper&lt;/a&gt;.&lt;/p&gt;

&lt;h4 id=&#34;softmax-output-layers&#34;&gt;Softmax output layers&lt;/h4&gt;

&lt;p&gt;Let \(r\) be the smallest perturbations to the input \(x\) of a softmax
layer such that \(f(x+r) \neq f(x)\). The authors have shown that this
condition is then satisfied:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/softmax.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;fully-connected-layers&#34;&gt;Fully connected layers&lt;/h4&gt;

&lt;p&gt;Here, the assumption is that the next layer has a robustness of \(k\). In that
case, the authors have shown that the minimum perturbations \(r\) satisfies
this condition:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/fulllayers.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;Here, \(J(x)\) is the Jacobian matrix of \(h_L\) at \(x\), and \(M\) is the
bound of the second order derivative of \(h_L\).&lt;/p&gt;

&lt;h4 id=&#34;convolutional-layers&#34;&gt;Convolutional layers&lt;/h4&gt;

&lt;p&gt;For a convolutional layer with filter tensor \(W \in R^{k \times c \times q
\times q}\), and input tensor \(X\), the adversarial perturbation \(R\)
satisfies this condition:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/convolutional.png&#34; width=&#34;200&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;pooling-layers&#34;&gt;Pooling layers&lt;/h4&gt;

&lt;p&gt;For a MAX or average pooling layer, the adversarial perturbation \(R\)
satisfies:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/maxpooling.png&#34; width=&#34;200&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;And for \(L_p\) pooling:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/lppooling.png&#34; width=&#34;200&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;h3 id=&#34;results&#34;&gt;Results&lt;/h3&gt;

&lt;p&gt;The authors conducted their experiments using the MNIST and CIFAR-10 datasets on
the LeNet network. For generating the adversarial perturbations for testing the
theoretical bounds, they used the fast gradient sign method (FGSM). They used a
binary search to find the smallest perturbation parameter of FGSM that resulted
in misclassification of a sample.&lt;/p&gt;

&lt;p&gt;In their experiments, the authors did not find any adversarial sample that
violated the theoretical lower bounds. The experimental and theoretical
perturbation norms can be seen in the following two tables:&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/fgsm.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Summary of norms of adversarial perturbations found by FGSM
&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class6/theoretical.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Summary of theoretical bound of adversarial perturbations
&lt;/p&gt;

&lt;p&gt;It can be seen that the mean theoretical perturbation is much lower than the
experimental ones for both MNIST and CIFAR-10 datasets. The authors suggest this
is due to FGSM not being the most efficient attacking technique.&lt;/p&gt;

&lt;p&gt;In conclusion, the authors suggest it is still unclear how tight these
theoretical bounds are. It will be interesting to verify how close they are to
the minimal distortion achieved by more efficient attacking techniques.
Moreover, they have only tried their method on feedforward networks. Applying it
to recurrent networks is a possible future direction. Finally, they propose the
adoption of a precise characterization of networks in terms of tradeoff between
robustness and accuracy.&lt;/p&gt;

&lt;p&gt;&amp;mdash; Team Bus:&lt;br /&gt;
Anant Kharkar,
Atallah Hezbor,
Bhuvanesh Murali,
Mainuddin Jonas,
Weilin Xu&lt;/p&gt;

&lt;h4 id=&#34;references&#34;&gt;References&lt;/h4&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1801.10578.pdf&#34;&gt;[1]&lt;/a&gt; T.-W Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su, Y. Gao, C.-J. Hsieh, L. Daniel &amp;ldquo;Evaluating the robustness of neural networks: An extreme value theory approach&amp;rdquo; &lt;em&gt;ICLR 2018&lt;/em&gt;, January 2018.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1608.04644.pdf&#34;&gt;[2]&lt;/a&gt; N. Carlini, D. Wagner &amp;ldquo;Towards evaluating the robustness of neural networks&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1608.04644&lt;/em&gt;, March 2017.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1312.6199.pdf&#34;&gt;[3]&lt;/a&gt; C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus. &amp;ldquo;Intriguing properties of neural networks&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1312.6199&lt;/em&gt;, 2013.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1705.08475.pdf&#34;&gt;[4]&lt;/a&gt; M. Hein, M. Andriushchenko. &amp;ldquo;Formal guarantees on the robustness of a classifier against adversarial manipulation&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1705.08475&lt;/em&gt;, 2017.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;[5]&lt;/a&gt; I.J. Goodfellow, J. Shlens, C. Szegedy. &amp;ldquo;Explaining and harnessing adversarial examples&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1412.6572&lt;/em&gt;, March 2015.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/abs/1709.10207&#34;&gt;[6]&lt;/a&gt; Nicholas Carlini, Guy Katz, Clark Barrett, David L. Dill &amp;ldquo;Provably Minimally-Distorted Adversarial Examples&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1709.10207&lt;/em&gt;, February 2018.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/abs/1702.01135&#34;&gt;[7]&lt;/a&gt; Guy Katz, Clark Barrett, David Dill, Kyle Julian, Mykel Kochenderfer &amp;ldquo;Reluplex: An efficient SMT solver for verifying deep neural networks.&amp;rdquo; &lt;em&gt;International Conference on Computer Aided Verification&lt;/em&gt;. Springer, Cham, 2017.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Class 5: Adversarial Machine Learning in Non-Image Domains</title>
      <link>https://secml.github.io/class5/</link>
      <pubDate>Fri, 23 Feb 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class5/</guid>
      <description>

&lt;h2 id=&#34;beyond-images&#34;&gt;Beyond Images&lt;/h2&gt;

&lt;p&gt;While the bulk of adversarial machine learning work has focused on image classification, ML is being used for a vartiety of tasks in the real world and attacks (and defenses) for different domains need to be tailored to the purpose of the ML process.
Among the most significant uses of ML are natural language processing and voice recognition. Within these fields, attacks look very different from those on images. With images, individual pixels can be changed large amounts without making the image unrecognizable, whereas in voice recognition, the audio needs to remain smooth, but many transformations can be done that are not noticable to humans.
These attacks are becoming more and more significant as always active listening devices like Amazon Alexa are becoming more common in the public sector. A broadcast television attack could be used to access personal accounts and devices of thousands of people simultaneously without many of those being attacked even noticing.&lt;/p&gt;

&lt;h2 id=&#34;adversarial-audio&#34;&gt;Adversarial Audio&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Nicholas Carlini and David Wagner. &lt;em&gt;Audio Adversarial Examples: Targeted Attacks on Speech-to-Text&lt;/em&gt;. University of California, Berkeley. 5 January 2018. [&lt;a href=&#34;https://arxiv.org/pdf/1801.01944.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If classic science fiction authors, painting grandiose visions of artificial intelligence in the future, could read some of the tech headlines of today, they might be surprised by how their predictions panned out:&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class5/headlines.png&#34; alt=&#34;&#34; title=&#34;Google Home and Alexa in the news&#34; /&gt;
  &lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://google.com)&#34;&gt;&lt;em&gt;Slides by Team Bus&lt;/em&gt;&lt;/a&gt;
   &lt;/div&gt;&lt;/p&gt;

&lt;p&gt;These news articles recount recent incidents in which smart home voice-controlled devices have been activated by voices in television advertisements and shows, rather than from a physically-present human voice. While these untargeted, accidental (we assume) occurrences perhaps make for amusing news, they reflect a significant potential vulnerability in speech-to-text systems against targeted, adversarial attacks.&lt;/p&gt;

&lt;p&gt;In their 2018 paper, authors &lt;a href=&#34;https://arxiv.org/pdf/1801.01944.pdf&#34;&gt;Carlini and Wagner&lt;/a&gt; develop and demonstrate a white-box attack against automated speech recognition systems: “given any audio waveform, we can produce another that is over 99.9% similar, but transcribes as any phrase of choice”. That is, a virtually unaltered sample, which might sound, at worst, noisy or mildly distorted to human ears, will be translated as an entirely different set of characters by the system.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class5/waveform_illustration.png&#34; alt=&#34;&#34; title=&#34;Illustration of the attack: adding small perturbations causes an audio waveform to transcribe to any desired target phrase.&#34; /&gt;
   &lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://arxiv.org/pdf/1801.01944.pdf)&#34;&gt;&lt;em&gt;Audio Adversarial Examples&lt;/em&gt;&lt;/a&gt; [1]
   &lt;/div&gt;&lt;/p&gt;

&lt;h4 id=&#34;audio-distortion&#34;&gt;Audio distortion&lt;/h4&gt;

&lt;p&gt;The goal of this attack is twofold: (a) to add the least possible amount of distortion to the original audio, and (b) to cause the distorted audio to map to an arbitrary sequence of transcribed characters. This thus becomes an optimization problem.&lt;/p&gt;

&lt;p&gt;To measure distortion, we use \(dB(v)\) to represent the loudness of a given sample \(v\) measured in decibels. First, let the change between the original and distorted samples be&lt;/p&gt;

&lt;p&gt;$$dB_x(\delta) := dB(\delta) - dB(x)$$&lt;/p&gt;

&lt;p&gt;where \(x\) is original and \(\delta\) distorted. Then, let&lt;/p&gt;

&lt;p&gt;$$C(v) = \mathop{\arg\,\max}\limits_p Pr[p | f(v)]$$&lt;/p&gt;

&lt;p&gt;be the most likely transcribed phrase given some sample \(v\).&lt;/p&gt;

&lt;p&gt;Our optimization problem becomes thus:&lt;/p&gt;

&lt;p&gt;$$\text{minimize } dB_x(\delta)$$
$$\text{such that } C(x+\delta)=t$$
$$\text{with } x+\delta \in [-M,M]$$&lt;/p&gt;

&lt;p&gt;where \(t\) is the target transcribed phrase and \(M\) is the maximum representable value (no distorted samples that are out of range &amp;ndash; in the authors&amp;rsquo; case, this happens to be \(2^{15}\) dB)).&lt;/p&gt;

&lt;p&gt;However, due to the nonlinearity of \(C(\cdot)\), gradient descent fails, so instead we minimize&lt;/p&gt;

&lt;p&gt;$$dB_x(\delta) + C(\cdot) \cdot \ell(x+\delta,t)$$
$$\text{with } \ell(x&amp;rsquo;,t) = \text{CTC-Loss}(x&amp;rsquo;,t)$$&lt;/p&gt;

&lt;p&gt;where the \(\text{CTC-Loss}\) (Connectionist Temporal Classification Loss) function is just the negative log-likelihood function.&lt;/p&gt;

&lt;p&gt;After solving this optimization problem, the authors constructed targeted examples on the first 100 test instances of Mozilla&amp;rsquo;s Common Voice dataset, attempting to distort 10 of these to achieve 10 target incorrect transcriptions &amp;ndash; with 100% success for source-target pairs, and a mean perturbation magnititude of -31 dB.&lt;/p&gt;

&lt;p&gt;Even after changing the loss function to&lt;/p&gt;

&lt;p&gt;$$\ell(x,pi) = \sum_i \max (f(x)^i_{\pi^i}-\max_{t&amp;rsquo;\neq \pi^i} f(x)^i_{t&amp;rsquo;},0) $$&lt;/p&gt;

&lt;p&gt;which avoids reducing loss and forces it to be more strongly classified, the mean distortion was only reduced from -31 dB to -38 dB, still a barely-noticeable to unnoticeable change.&lt;/p&gt;

&lt;h4 id=&#34;attack-sources-and-targets&#34;&gt;Attack Sources and Targets&lt;/h4&gt;

&lt;p&gt;Attack models worked consistently regardless of context, meaning that an attack using white noise or non-speech is as easy to craft as one using speech. Targeting silence turned out to be even easier than targeting speech.&lt;/p&gt;

&lt;h4 id=&#34;robustness&#34;&gt;Robustness&lt;/h4&gt;

&lt;p&gt;Random pointwise noise of less than -30dB is sufficient to corrupt an attack; although stronger attacks can be crafted using larger dB changes. The attacks did manage to be MP3 resistent, which speaks to the minimization of loss in MP3 compression as well as the fidelity of the attack model.&lt;/p&gt;

&lt;p&gt;These attacks do become easier with longer sources, as the pacing of the target speech can be varied more, giving a larger attack space in which to work. In an ideal case for an attacker, they would be working with a prerecorded broadcast, which gives them a large continuous space to attack.
That said, Over-The-Air attacks were not viable due to the white noise and variation in speakers and the rooms. Hidden voice commands did, however, show some promise for broadcast attacks.&lt;/p&gt;

&lt;p&gt;Transferability was shown to hold, as it is somewhat fundamental to ML systems. As of yet, defence techniques on the ML layers have not yet been developed, although random noise added post-attack is a viable defence.&lt;/p&gt;

&lt;h2 id=&#34;natural-language-processing&#34;&gt;Natural Language Processing&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Suranjana Samanta and Sameep Mehta. &lt;a href=&#34;https://arxiv.org/pdf/1707.02812.pdf&#34;&gt;Towards Crafting Text Adversarial Samples&lt;/a&gt;. 2017.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Adversarial samples are strategically modified samples, which are crafted with the purpose of fooling a classifier at hand. Although most of the prior works have been focused on synthesizing adversarial samples in the image domain, NLP based text classifiers can also be the focal point of such exploit. This paper introduces a method of crafting adversarial text samples by modification of the original samples. To be precise, the authors propose to modify the original text samples by deleting or replacing the important  words in the text or by introducing new words in the text sample. While crafting adversarial samples, the paper focuses on generating meaningful sentences which can pass off as legitimate from language viewpoint.&lt;/p&gt;

&lt;h3 id=&#34;approach&#34;&gt;Approach&lt;/h3&gt;

&lt;p&gt;Initially, the authors calculate contribution of each word towards determining the class-label.  A word in the text is highly contributing if its removal from the text is going to change the class probability value to a large extent. In their proposed approach, modification of the sample text by considering each word at a time, in the order of the ranking based on the class-contribution factor. For the modification purpose, a candidate pool for each word in sample text is created considering synonyms and typos of each of the words as well as the genre or sub-category specific keywords. The assumption behind choosing sub-category or genre-specific keywords based on the fact that certain words may contribute to positive sentiment for a particular genre but can emphasis negative sentiment for other kind of genre. For example, we can consider the sentence, &amp;ldquo;The movie was hilarious&amp;rdquo;. This indicates a positive sentiment for a comedy movie. But the same sentence denotes a negative sentiment for a horror movie.  Thus, the word &amp;lsquo;hilarious&amp;rsquo; contributes to the sentiment of the review based on the genre of the movie. Finally, replacement, addition or removal of words are performed on a given text sample at each iteration, so that the modified sample flips its class label.&lt;/p&gt;

&lt;h3 id=&#34;experiments&#34;&gt;Experiments&lt;/h3&gt;

&lt;p&gt;The paper performs their experimental results on two datasets: IMDB movie review dataset for sentiment analysis and twitter dataset for gender classification. They compare the efficiency of their method with the existing method TextFool by measuring the accuracy of the model
obtained at different configurations. For both the cases, the proposed model was generated using Convolutional Neural Network (CNN).&lt;/p&gt;

&lt;p&gt;The figure below (taken from the paper) shows the results for IMDB dataset. From the figure, it is obvious that the proposed method of adversarial sample crafting for text is capable of synthesizing semantically correct adversarial text samples from the original text sample. In addition, the inclusion of genre specific keywords appears to increase the quality of sample crafting. This is evident from the fact the drop in accuracy of the classifier before re-training for original text sample and the adversarialy crafted text sample is more when genre specific keywords are being used.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/result_imdb.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt;  &lt;b&gt;Figure:&lt;/b&gt; Performance results on IMDB movie review dataset.
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;A significant factor for evaluating the effectiveness of adversarial samples is to measure the semantic similarity between the original samples and their corresponding tainted counterparts. A lower similarity score denotes that the semantic meaning of the original and the modified samples are quite different, which is not acceptable from the language viewpoint. The authors found the average semantic similarity between the original text sample and their adversarial counterparts (for test set only) as 0.9164 and 0.9732 with and without using the genre specific keywords respectively. Although the semantic similarity between the original and their corresponding perturbed samples does decrease a bit while the genre specific keywords in the candidate pool, the number of valid adversarial samples generated increases as it can be seen from the percentage of the perturbed samples for genre specific keywords in the above figure.&lt;/p&gt;

&lt;p&gt;Another key component for the evaluation of adversarial samples is to measure the number of changes incurred to obtain the adversarial sample. The number of changes made to craft a successful adversarial sample should be ideally low.The following figure shows a graph that indicates the number of changes required to create successful adversarial samples with and without using the genre-specific keywords. From the figure, it is evident that, if we consider same number of modifications the number of tainted sample produced using genre specific keywords for creating adversarial samples is more than that when genre specific keywords are not used.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/result_modifications.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt;  &lt;b&gt;Figure:&lt;/b&gt; Plot showing the number of adversarial samples produced against the number of changes incurred
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;For the Twitter dataset, the proposed method shows similar performance as the IMDB dataset.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/result_twitter.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt;  &lt;b&gt;Figure:&lt;/b&gt; Performance results on Twitter gender classification dataset.
&lt;/p&gt;&lt;/p&gt;

&lt;h3 id=&#34;example&#34;&gt;Example&lt;/h3&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/example_nlp.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt;  &lt;b&gt;Figure:&lt;/b&gt; Examples of adversarial samples crafted from Twitter and IMDB dataset using (i)TextFool and
(ii) proposed method.
&lt;/p&gt;

&lt;h4 id=&#34;face-recognition&#34;&gt;Face Recognition&lt;/h4&gt;

&lt;blockquote&gt;
&lt;p&gt;Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, Michael K. Reiter. &lt;a href=&#34;https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf&#34;&gt;&lt;em&gt;Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition&lt;/em&gt;&lt;/a&gt;. ACM CCS 2016.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Face recognition and face detection algorithms are extensively used in surveillance and access control applications. This paper focuses on fooling the state-of-art machine learning algorithms that are practically deployed for these tasks. More concretely, the paper carries out two types of attacks: dodging and impersonation. In dodging, the attacker tries to conceal his/her identity, whereas, in impersonation, the attacker tries to trick the algorithm into recognizing him/her as a different target individual. The authors realize these attacks by printing a wearable glass which, upon wearing, allows the attacker to successfully launch the attacks. The glasses used in the attack are shown below.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class5/glasses.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Glass frames used for the attacks
&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.robots.ox.ac.uk/~vgg/publications/2015/Parkhi15/parkhi15.pdf&#34;&gt;Parikh et al.&lt;/a&gt; proposed the state-of-art 39 layer deep neural network trained on 2,622 celebrities which achieved an accuracy of 98.95% for the task of face detection. The authors of this paper use this model to launch their attacks.&lt;/p&gt;

&lt;p&gt;The objective function of the deep neural network&amp;rsquo;s softmax layer is given as below:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class5/objective.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;For impersonation, the objective is to add the minimum amount of noise \(r\) in the input image \(x\) to convert the class lable to the target label \(c_t\), as given below:
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class5/impersonation_obj.png&#34; width=&#34;450&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;For dodging, the objective is to add minimum amount of noise \(r\) in the input image \(x\) to deviate from the correct class label \(c_x\).
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class5/dodging_obj.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;The noises are then carefully added to the 3D printed glasses.&lt;/p&gt;

&lt;h4 id=&#34;experiments-1&#34;&gt;Experiments&lt;/h4&gt;

&lt;p&gt;The paper launches successful dodging and impersonation attacks. Figure below shows an example of impersonation where a female celebrity (left) is classified as a male celebrity (right) by the algorithm when the glass frame is used (center).
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class5/example1.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Successful impersonation using the glasses
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;Figure below shows an example of dodging the face detection algorithm with minor changes to the image that donot affect a normal human judgement. Left image is the original image and the center and right images are the perturbed images where the algorithm does not detect the faces.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class5/example2.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Dodging the face detection
&lt;/p&gt;&lt;/p&gt;

&lt;p&gt;Further results show 100% dodging success rate and very high impersonation success rate.&lt;/p&gt;

&lt;h4 id=&#34;summary&#34;&gt;Summary&lt;/h4&gt;

&lt;p&gt;While the authors perform white-box attacks on the state-of-the-art face detection and face recognition algorithms, they also discuss about successfully carrying out the black-box attacks.&lt;/p&gt;

&lt;p&gt;— Team Nematode: &lt;br /&gt;
Bargav Jayaraman, Guy &amp;ldquo;Jack&amp;rdquo; Verrier, Joshua Holtzman, Max Naylor, Nan Yang, Tanmoy Sen&lt;/p&gt;

&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1801.01944.pdf&#34;&gt;[1]&lt;/a&gt; Nicholas Carlini and David Wagner, &amp;ldquo;Audio Adversarial Examples: Targeted Attacks on Speech-to-Text.&amp;rdquo; January 2018.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cs.cmu.edu/~sbhagava/papers/face-rec-ccs16.pdf&#34;&gt;[2]&lt;/a&gt; Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter, &amp;ldquo;Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition.&amp;rdquo; October 2016.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Class 4: Differential Privacy In Action</title>
      <link>https://secml.github.io/class4/</link>
      <pubDate>Thu, 22 Feb 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class4/</guid>
      <description>

&lt;p&gt;Two weeks ago we took a look at &lt;a href=&#34;https://secml.github.io/class2/&#34;&gt;privacy in machine learning&lt;/a&gt; and introduced differential privacy as one possible approach to perform statistical analysis on data while maintaining user privacy.  Today we explore three applications of differential privacy: Google&amp;rsquo;s RAPPOR for obtaining user data from client-side software, the FLEX system to enforce differential privacy for SQL queries, and an algorithm for training deep neural networks that can provide differential privacy guarantees.&lt;/p&gt;

&lt;h2 id=&#34;google-s-rappor&#34;&gt;Google&amp;rsquo;s RAPPOR&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Erlingsson, Úlfar, Vasyl Pihur, and Aleksandra Korolova. &lt;em&gt;Rappor: Randomized aggregatable privacy-preserving ordinal response&lt;/em&gt;. ACM CCS 2014. [&lt;a href=&#34;https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42852.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The goal of Randomized Aggregatable Privacy-Preserving Ordinal Response (RAPPOR) is to ensure anonymity for those participating in crowd-sourced statistics with a strong privacy guarantee. For example, if student who has cheated is participating in a study on cheating in school it is unlikely they would respond truthfully without strong plausible deniability.&lt;/p&gt;

&lt;p&gt;A simple algorithm that constructs such plausible deniability is shown below:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;STEP ONE

flip coin
if (coin==HEADS):
    answer truthfully
else:
    go to step 2

STEP TWO

flip coin
if (coin==HEADS):
    answer yes
else:
    answer truthfully
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This algorithm will result in a truthful response rate of 75%. Therefore if we let \(Y = [\text{Number of raised hands}] / [\text{Size of class}]\) then \([\text{Number of true cheaters}] \approx 2(Y − 0.25)\).&lt;/p&gt;

&lt;p&gt;This means we have differential privacy with the level \(\epsilon = \ln(0.75/(1 − 0.75)) = \ln(3)\), but only for the first time responses are collected. This guarantee degrades if the same survey is administered repeatedly (where the same respondants generate new randomness for each query). RAPPOR is a way to ensure strong privacy protection even for a single respondent who is surveyed often.&lt;/p&gt;

&lt;p&gt;On a high level, RAPPOR achieves this by having each client machine report a “noisy” representation of the true value \(v\) by submitting a \(k\)-sized bit array to a server. This representation of \(v\) is selected in order to reveal a specific amount of information about \(v\) in order to limit the information the server learns from the \(k\)-sized bit array. Importantly the server does not learn the true value \(v\) with confidence even when an infinite number of reports are submitted by the client.&lt;/p&gt;

&lt;p&gt;This is achieved through the the following three steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;em&gt;Signal&lt;/em&gt; - hash the client’s value \(v\) onto the Bloom filter \(B\) of size \(k\) using \(h\) hash functions.&lt;/li&gt;
&lt;/ol&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/rappor_signal.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Bloom Filter
&lt;/p&gt;

&lt;p&gt;A Bloom filter is a probabilistic data structure optimized for determining set membership.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;em&gt;Permanent&lt;/em&gt; Randomized Response
For each value \(v\) from a client, and for each bit \(i\) in the bloom filter \(B\), create a binary reporting value \(B_i’\) such that TODO:missing equation here??? where \(f\) is a user-defined tuning variable, \(f \in (0,1)\). This \(B’\) is memorized and reused as the basis for all future reports on value \(v\).&lt;/li&gt;
&lt;/ol&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/rappor_perm_prob.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Permanent B&#39;
&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;em&gt;Instantaneous Randomized Response&lt;/em&gt;. Next, allocate a bit array \(S\) of size \(k\) and initialize to 0. Set each bit \(i\) in \(S\) with the following probabilities: TODO:???&lt;/li&gt;
&lt;/ol&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/rappor_inst_prob.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;em&gt;Report&lt;/em&gt;. Send the bit array \(S\) to the server.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;How private are RAPPOR’s aggregated results in reality? If you assume infinite sampling, there is a privacy guarantee of:&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/rappor_epsilon_inf.png&#34; width=&#34;500&#34; &gt;
&lt;br&gt;&lt;/p&gt;

&lt;p&gt;An attacker with &amp;ldquo;unlimited collection capabilities&amp;hellip; is also bounded by the privacy guarantee of ε∞ and cannot improve upon this bound with more data collection.&amp;rdquo;&lt;/p&gt;

&lt;h2 id=&#34;towards-practical-differential-privacy-for-sql-queries&#34;&gt;Towards Practical Differential Privacy for SQL Queries&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Johnson, Noah, Joseph P. Near, and Dawn Song. &lt;em&gt;Towards Practical Differential Privacy for SQL Queries&lt;/em&gt;. Proceedings of the VLDB Endowment, January 2018. &lt;a href=&#34;Images below are taken from this paper.&#34;&gt;&lt;a href=&#34;https://arxiv.org/pdf/1706.09479.pdf&#34;&gt;PDF&lt;/a&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h4 id=&#34;motivation&#34;&gt;Motivation&lt;/h4&gt;

&lt;p&gt;The recent increase in data collection in large organizations has exposed personal user data to analysis that have an impact on user privacy. This paper investigates the methods that can be used to increase differential privacy, protecting individual privacy. By taking a practical approach to differential privacy, the authors are able to provide an implementation of differential privacy that is compatible with any existing database.&lt;/p&gt;

&lt;h4 id=&#34;contributions&#34;&gt;Contributions&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;Empirical study of 8.1 million real-world SQL queries&lt;/li&gt;
&lt;li&gt;Elastic sensitivity&lt;/li&gt;
&lt;li&gt;FLEX: An end-to-end differential privacy system&lt;/li&gt;
&lt;li&gt;Experimental evaluation of FLEX&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&#34;elastic-sensitivity&#34;&gt;Elastic sensitivity&lt;/h4&gt;

&lt;p&gt;Elastic sensitivity is the paper’s main contribution and proposed approach to enforcing differential privacy on a system that is locally sensitivity-based. Local sensitivity is the “maximum of the difference between the query run on a true database and any neighbor of it.” The practical nature of their design exists because instead of being a property of all possible databases, local sensitivity is a property of one true database.
&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/ls.png&#34; width=&#34;300&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Local Sensitivity
&lt;/p&gt;
The paper describes and proves their theorem shown below using aspects of database local sensitivity.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/proof.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Elastic Sensitivity
&lt;/p&gt;

&lt;h4 id=&#34;limitation-of-existing-approaches&#34;&gt;Limitation of existing approaches&lt;/h4&gt;

&lt;p&gt;Two main requirements of differential privacy are stated. The first is that the implementation of differential privacy must be compatible with existing databases, and have support for heterogeneous databases. Also it should have robust support for equijoins, which include self and non-self joins, relationship types and an arbitrary number of nested joins.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/g1.PNG&#34; width=&#34;600&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Comparison of various differential privacy implementations
&lt;/p&gt;

&lt;p&gt;As seen in the chart above, other approaches address parts of the requirements, and elastic sensitivity seems to cover all of the requirements set forth in this paper for differential privacy. One lacking requirement among previous approaches was adequate database compatibility.&lt;/p&gt;

&lt;h4 id=&#34;unsupported-queries-and-other-aggregate-functions&#34;&gt;Unsupported Queries and other aggregate functions&lt;/h4&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/sqlcode.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; SQL non-equijoin statement
&lt;/p&gt;

&lt;p&gt;Since non-equijoins need information about both datasets, the operation is not supported in elastic sensitivity. Elastic sensitivity also sometimes fails to provide support for max-frequency metrics. However, not having support for either of these operations does not impact a majority of operations in either case.&lt;/p&gt;

&lt;h4 id=&#34;flex&#34;&gt;FLEX&lt;/h4&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/flexPerf.PNG&#34; width=&#34;500&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; FLEX design structure
&lt;/p&gt;

&lt;p&gt;FLEX is an implementation of elastic sensitivity that is highly compatible with existing databases. The FLEX database structure is shown in the figure above. TODO: this is the wrong figure&lt;/p&gt;

&lt;p&gt;By targeting support for specific SQL queries, their system can enforce differential privacy on most real world queries to databases. For a given SQL query, FLEX calculates its elastic sensitivity given an analysis of the query. FLEX then applies smooth sensitivity to the elastic sensitivity and adds noise drawn from the Laplace distribution to the original query results.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/flexPerf.PNG&#34; width=&#34;600&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; FLEX Performance
&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Elastic sensitivity for 76% of queries&lt;/li&gt;
&lt;li&gt;Large error for unsupported queries: 14.14%&lt;/li&gt;
&lt;li&gt;Parsing errors: 6.58%&lt;/li&gt;
&lt;/ul&gt;

&lt;h4 id=&#34;comparison-with-wpinq&#34;&gt;Comparison with wPINQ&lt;/h4&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/wPINQComp.PNG&#34; width=&#34;1000&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; wPINQ and Flex performance comparison
&lt;/p&gt;

&lt;p&gt;This paper moves towards more practical approaches of differential privacy in SQL queries by proposing elastic sensitivity. The concept of elastic sensitivity was then tested through their implementation in FLEX. A comparison of FLEX and wPINQ is shown in the figure above. Adoption of this method by Uber for internal data analytics demonstrates the potential of their approach for having a large impact on data privacy.&lt;/p&gt;

&lt;h2 id=&#34;deep-learning-with-differential-privacy&#34;&gt;Deep learning with differential privacy&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kinal Talwar, and Li Zhang. &lt;em&gt;Deep learning with differential privacy&lt;/em&gt;. ACM CCS 2016 &lt;a href=&#34;All figures below are taken from this paper&#34;&gt;&lt;a href=&#34;https://arxiv.org/pdf/1607.00133.pdf&#34;&gt;PDF&lt;/a&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h4 id=&#34;differential-privacy-recap&#34;&gt;Differential Privacy Recap&lt;/h4&gt;

&lt;p&gt;Differential Privacy can be defined in terms of the application-specific concept of adjacent databases. Suppose, for adjacent databases where each training dataset contains a set of image-label pairs, we say that two of these sets are adjacent if one image-label pair is present in one training set while absent in the other. A randomized mechanism \( \mathcal{M}: D \rightarrow R\) with domain D and range R satisfies \( (\epsilon, \delta) \)-differential privacy if for any two adjacent inputs \( d,d&amp;rsquo;~\epsilon ~D\) and for any subset of outputs \( S \subseteq R\) it holds that&lt;/p&gt;

&lt;p&gt;$$ Pr[\mathcal{M}(d) ~\epsilon ~\mathcal{S}] \leq \mathcal{e}^{\epsilon} Pr[\mathcal{M}(d&amp;rsquo;) ~\epsilon ~\mathcal{S}] + \delta $$&lt;/p&gt;

&lt;p&gt;Authors used Dwork et al.[1] privacy definition of allowing the possibility that plain \( \epsilon \)-differential privacy is broken with probability \( \delta \) in their work.&lt;/p&gt;

&lt;h2 id=&#34;approach&#34;&gt;Approach&lt;/h2&gt;

&lt;p&gt;Their proposed technique consists of three components: differentially private SGD (Stochastic Gradient Descent) Algorithm, Moments Accountant and Hyperparameter Tuning.&lt;/p&gt;

&lt;h4 id=&#34;differentially-private-sgd-algorithm&#34;&gt;Differentially private SGD Algorithm&lt;/h4&gt;

&lt;p&gt;Algorithm 1 describes their method for training a model with parameters \( \theta \) by minimizing the loss function \( L(\theta) \). At each step of computing of SGD, they calculated the gradient \( \nabla \theta L(\theta, x_i)\) for a random subset then clip the \(L_2\) norm of each gradient. After that they computed the average and for ensuring the privacy they added noise in those. They took the opposite direction of this average noisy gradient. Finally, they ouput the model with the privacy loss.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/algorithm_diff_priv_sgd.png&#34; width=&#34;600&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Code snippet of Differentially private SGD
&lt;/p&gt;

&lt;h4 id=&#34;moments-accountant&#34;&gt;Moments Accountant&lt;/h4&gt;

&lt;p&gt;Each lot is \( (\epsilon, \delta)\)-DP if we choose \(\sigma \) in Algorithm 1 as \( \sigma \) = \(\frac{\sqrt{ 2\log(\frac{1.5}{\delta})}} {\epsilon} \) for Gaussian noise. Thus, each step is \( \mathcal{O}((q \epsilon),q\delta) \)-DP over the dataset, where q = \( \frac{L}{N}\) is the sampling probability of a lot over the dataset and \(\epsilon \leq 1 \).
The result in the literature which gives the best overall bound is the strong composition theorem [2]. Strong composition theorem does not take into account any particular noise distribution under consideration. Authors invent a stronger accounting method, which is the moments accountant. In Algorithm 1, over \(T\) iterations, naive composition gives the bound of \( \mathcal{O}((qT \epsilon),qT\delta) \)-DP. Over \(T\) iterations, strong composition gives the bound of \( \mathcal{O}((q \epsilon \sqrt{T \log \frac{1}{\delta}}),qT\delta) \)-DP. Whereas, over \(T\) iterations, moments accountant gives a tighter bound of \( \mathcal{O}((q \epsilon \sqrt{T}),\delta) \)-DP.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Theorem:&lt;/strong&gt; There exist constants \(c_1\) and \(c_2\) so that given the sampling probability q = L/N and the number of steps T, for any \( \epsilon &amp;lt; c_1q^{2}T \), Algorithm 1 is \( (\epsilon,\delta) \)-differentially private for any \(\delta &amp;gt; 0\) if we choose $$ \sigma \geq c_2 \frac{q \sqrt{T\log(\frac{1}{\delta})}}{\epsilon} $$&lt;/p&gt;

&lt;p&gt;If we use the strong composition theorem, we will then need to choose \( \sigma = \Omega(q\sqrt{T\log(1/\delta)\log(T/\delta)}/\epsilon)\)
For \(L = 0.01N\), \(\sigma\) = 4, \(\delta\) = \(10^{-5}\), and \(T = 10000\), we have \(\epsilon\) ≈ 1.26 using the moments accountant, and \(\epsilon\) ≈ 9.34 using the strong composition theorem.&lt;/p&gt;

&lt;h4 id=&#34;hyperparameter-tuning&#34;&gt;Hyperparameter Tuning&lt;/h4&gt;

&lt;p&gt;Authors used the insight from the version of a result from Gupta et al. [3] restated as Theorem D.1 in the Appendix to reduce the number of hyperparameter settings. From theorem 1, one can say that making batches size too large will increase the privacy cost. The learning rate in non-private training will also go downwards as the model will converge to a local optimum. But authors found that they didn&amp;rsquo;t have to decrease the learning rate to a very small value because differentially private training never goes to an area where it would be justified. Moreover, from their experiment they came to know that there was a small advantage of starting with a large learning rate compare to small learning rate and then linearly decaying it to a smaller value in a few epochs. And after that the rate remained constant.&lt;/p&gt;

&lt;h2 id=&#34;implementation&#34;&gt;Implementation&lt;/h2&gt;

&lt;p&gt;They have implemented the differentially private SGD algorithms in TensorFlow. They made their code available to download under an Apache 2.0 license from github.com/tensorflow/models. Their whole implementation is divided into two components. One is Sanitizer which preprocesses the gradient to protect privacy and the other one is privacy accountant which keeps track of the privacy cost of training the model. They implemented differentially private PCA(Principal Directions) and applied pre-trained convolutional layers. Because the neural network model may benefit by projecting the inputs on the PCA or by feeding it through a convolutional layer.&lt;/p&gt;

&lt;h4 id=&#34;sanitizer&#34;&gt;Sanitizer&lt;/h4&gt;

&lt;p&gt;For achieving the privacy protection, the sanitizer performs the following two operations:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Limits sensitivity of each example by clipping the norm of the gradient&lt;/li&gt;
&lt;li&gt;Adds noise to the gradient of a batch before updating network parameters&lt;/li&gt;
&lt;/ol&gt;

&lt;h4 id=&#34;privacy-accountant&#34;&gt;Privacy Accountant&lt;/h4&gt;

&lt;p&gt;The main purpose of their implementation is to keep track of privacy spending over the course of training and hence the implementation of Privacy Accountant is the most important component of their implementation. One can compute \(\alpha (\gamma)\) by two ways. First one is by applying asymptotic bound and then evaluating a closed-form expression and the second one is by applying numerical integration. In their implementation they used the second one to compute \(\alpha (\gamma)\). They also \(\alpha (\gamma)\) computed for a wide range of \( \gamma&amp;rsquo;s\), which helped them to get the best result possible \( (\epsilon, \delta) \) values. They stated that for the parameters it is enough to calculate \(\alpha (\gamma)\) where \(\gamma \leq 32\).&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/algorithm_dpsgd_optimizer.png&#34; width=&#34;600&#34; &gt;
&lt;br&gt; &lt;b&gt;Figure:&lt;/b&gt; Code snippet of DPSGD_Optimizer and DPTrain
&lt;/p&gt; 

&lt;p&gt;The above figure contains the TensorFlow code snippet (in Python) of DPSGD_Optimizer which iteratively invokes DPSGD_Optimizer using a privacy accountant to bound the total privacy loss.&lt;/p&gt;

&lt;h4 id=&#34;differentially-private-pca&#34;&gt;Differentially private PCA&lt;/h4&gt;

&lt;p&gt;Differentially Private Principal Component Analysis is a very useful method for capturing the features of the input data. Authors implemented the differentially private PCA algorithm according to Dwork et al.[4]. They took a random sample from the training examples and treated them as vectors. They normalized each vector to unit \(l_2\) norm to form the matrix \(A^{T}A\). After that they added Gaussian noise to the matrix A. Then for each input example they applied the projection to the principal directions of noisy covariance matrix before feeding them into the neural network.&lt;/p&gt;

&lt;h2 id=&#34;experimental-results&#34;&gt;Experimental Results&lt;/h2&gt;

&lt;h4 id=&#34;mnist&#34;&gt;MNIST&lt;/h4&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/result_accuracy_mnist_fig_3.png&#34;&gt;
&lt;/p&gt; 

&lt;p&gt;&lt;strong&gt;Figure:&lt;/strong&gt; Results on the accuracy for different noise levels on the MNIST dataset. In all the experiments, the network uses 60 dimension PCA projection, 1,000 hidden units, and is trained using lot size 600 and clipping threshold 4. The noise levels \(\sigma, \sigma_{p}\) for training the neural network and for PCA projection are set at (8, 16), (4, 7), and (2, 4), respectively, for the three experiments.&lt;/p&gt;

&lt;p&gt;In the above Figure, they showed the results for different noise levels. In
each plot, they showed the evolution of the training and testing
accuracy as a function of the number of epochs as well as
the corresponding \(\delta\) value, keeping \(\epsilon\) fixed.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/result_accuracy_mnist_fig_4.png&#34; width=&#34;600&#34; &gt;
&lt;/p&gt; 

&lt;p&gt;&lt;strong&gt;Figure:&lt;/strong&gt; Accuracy of various \( (\epsilon,\delta) \) privacy values
on the MNIST dataset. Each curve corresponds to a different \(\delta \) value.&lt;/p&gt;

&lt;p&gt;In the above figure, they showed the accuracy that they can obtain for different value of \(\delta\) and \(\epsilon\). From the figure, one can observe that keeping the value of \(\delta\) fixed and varying the value of \(\epsilon\) can have large impact on the accuracy.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/result_accuracy_mnist_fig_5.png&#34; &gt;
&lt;/p&gt; 

&lt;p&gt;&lt;strong&gt;Figure:&lt;/strong&gt; MNIST accuracy when one parameter varies, and the others are fixed at reference values.&lt;/p&gt;

&lt;p&gt;In the above figure, they showed the accuracy of the model on MNIST dataset by by varying a particular parameter while keeping the other parameter constant. For training the neural network parameters and for PCA projection they set the reference values like - 60 PCA dimenstions, 600 lot size, 1000 hidden units, initial learning rate of 0.1, final learning rate 0.052 in 10 epochs, gradient norm bound of 4, and noise equal to 4 and 7 respectively.&lt;/p&gt;

&lt;h4 id=&#34;cifar-10&#34;&gt;CIFAR-10&lt;/h4&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/class4/result_accuracy_cifar_fig_6.png&#34; &gt;
&lt;/p&gt; 

&lt;p&gt;&lt;strong&gt;Figure:&lt;/strong&gt; Results on accuracy for different noise levels on CIFAR-10. With \(\delta \) set to \(10^{-5}\) , achieve accuracy 67%, 70%, and 73%, with \(\epsilon \) being 2, 4, and 8, respectively. The first graph uses a lot size of 2,000, (2) and (3) use a lot size of 4,000. In all cases, \(\sigma \) is set to 6, and clipping is set to 3.&lt;/p&gt;

&lt;p&gt;In the above figure, authors showed the result of the accuracy and privacy cost of the model on the CIFAR-10 dataset as a function of the number of epochs for different parameter settings. In MNIST dataset, the difference in accuracy between a non-private baseline and a private model is about 1.3% whereas the corresponding drop in accuracy in CIFAR-10 dataset is about 7%.&lt;/p&gt;

&lt;p&gt;&amp;mdash; Team Panda
Christopher Geier, Faysal Hossain Shezan, Helen Simecek, Lawrence Hook, Nishant Jha&lt;/p&gt;

&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;

&lt;p&gt;&lt;a href=&#34;http://www.wisdom.weizmann.ac.il/~naor/PAPERS/odo.pdf&#34;&gt;[1]&lt;/a&gt; C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov,and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486–503. Springer, 2006.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://privacytools.seas.harvard.edu/files/privacytools/files/05670947.pdf&#34;&gt;[2]&lt;/a&gt; C. Dwork, G. N. Rothblum, and S. Vadhan. Boosting and differential privacy. In FOCS, pages 51–60. IEEE, 2010.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/0903.4510.pdf&#34;&gt;[3]&lt;/a&gt; A. Gupta, K. Ligett, F. McSherry, A. Roth, and K. Talwar. Differentially private combinatorial optimization. In SODA, pages 1106–1125, 2010.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;http://kunaltalwar.org/papers/PrivatePCA.pdf&#34;&gt;[4]&lt;/a&gt; C. Dwork, K. Talwar, A. Thakurta, and L. Zhang. Analyze Gauss: Optimal bounds for privacy-preserving principal component analysis. In STOC, pages 11–20. ACM, 2014.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Topic Suggestions</title>
      <link>https://secml.github.io/topic-suggestions/</link>
      <pubDate>Tue, 20 Feb 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/topic-suggestions/</guid>
      <description>&lt;p&gt;To help with topics for future classes, I&amp;rsquo;ve created a
&lt;a href=&#34;https://secml.github.io/topics&#34;&gt;Topics&lt;/a&gt; page with some possible ideas and links to papers.
This is not meant to be exhaustive by any stretch; any topics loosely
connected to machine learning security and privacy is within scope,
and there are lots of great papers on these topics not included on
this page.  But, hopefully it will be a useful starting point for
teams looking for ideas for topics for future classes.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Class 3: Adversarial Machine Learning</title>
      <link>https://secml.github.io/class3/</link>
      <pubDate>Fri, 09 Feb 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class3/</guid>
      <description>

&lt;p&gt;This week’s topic covered some proposed adversarial example attacks and defenses. The underlying problem is that machine learning techniques assume that training and testing data are generated from the same distribution. Therefore, adversaries can choose inputs to exploit the algorithms by manipulating data. We began class by discussing common distance metrics,
\(L_0, L_2\), and \(L\infty\), popular benchmarking datasets, and the history of adversarial ML. However, the main theme was defense techniques can be used safely to prevent adversarial attacks. Below we discuss four papers that discuss both effective and ineffective defenses.&lt;/p&gt;

&lt;h2 id=&#34;distillation-as-a-defense&#34;&gt;Distillation as a Defense&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Nicolas Papernot and
               Patrick D. McDaniel,
               Xi Wu,
               Somesh Jha, and
               Ananthram Swami. &lt;em&gt;Distillation as a Defense to Adversarial Perturbations against Deep
               Neural Networks&lt;/em&gt;. IEEE Symposium on Security and Privacy, 2016. [&lt;a href=&#34;https://arxiv.org/abs/1511.04508&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3 id=&#34;what-is-distillation&#34;&gt;What is Distillation&lt;/h3&gt;

&lt;p&gt;Neural networks typically produce class probabilities by using a “softmax” output layer that converts the logit, \(z_i\), computed for each class into a probability, \(q_i\), by comparing \(z_i\) with the other logits.&lt;/p&gt;

&lt;p&gt;$$ q_i = \frac{\exp(z_i/T)}{\sum_j exp(z_j/T)} $$&lt;/p&gt;

&lt;p&gt;where \(T\) is a temperature that is normally set to 1. Using a higher value for \(T\) produces a softer probability distribution over classes.&lt;/p&gt;

&lt;p&gt;In the simplest form of distillation, knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax. The same high temperature is used when training the distilled model, but after it has been trained it uses a temperature of 1. It could reduce the computing resources required to run a network, allowing usage on a smaller scale like in embedded chips and IoT devices.&lt;/p&gt;

&lt;h3 id=&#34;how-does-distillation-work&#34;&gt;How does Distillation Work?&lt;/h3&gt;

&lt;p&gt;1) A Deep Neural Network(DNN) is trained with a high temperature, the \(T\) we mentioned before. The training of this first DNN is a high temperature because the high temperature forces the DNN to produce probability vectors with relatively large values for each class. The high temperature of a softmax is, the more ambiguous its probability distribution will be. The smaller the temperature of a softmax is, the more discrete its probability distribution will be.&lt;/p&gt;

&lt;p&gt;2) A second Deep Neural Network is trained by replacing the hard labels of the training set with class probabilities output by the first Deep Neural Network.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/DNN.png&#34; alt=&#34;&#34; /&gt;
&lt;div class=&#34;caption&#34;&gt;TODO: add source&lt;/div&gt;&lt;/p&gt;

&lt;h3 id=&#34;softmax-function-under-distillation&#34;&gt;Softmax Function under distillation&lt;/h3&gt;

&lt;p&gt;Softmax function is the Last layer of network. It’s used to normalize the outputs of the second to last layer. Under distillation situation, it has a parameter temperature (\(T\)).
To perform distillation in softmax layer, a large network whose output layer is softmax is first trained on the original dataset. The softmax layer is a layer that considers a vector \(Z(X)\) of outputs produced by the last hidden layer of a DNN. Then we normalizes them into a probability vector \(F(X)\), the output of DNN assigning a probability to each class of dataset for input \(X\). \(T\) means temperature and shared across the softmax layer.  (See the paper for the equations for \(F(X)\).)&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/softmax.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;h3 id=&#34;using-distillation-as-a-defense&#34;&gt;Using Distillation as a Defense&lt;/h3&gt;

&lt;p&gt;In distillation as a defense, the same network architecture is used in the distilled DNN as in the original DNN. First, this paper trained an initial network \(F\) on data \(X\) with a softmax temperature of \(T\).&lt;/p&gt;

&lt;p&gt;Then, this paper uses the probability vector \(F(X)\), which includes additional knowledge about classes compared to a class label, predicted by network \(F\) to train a distilled network  at temperature \(T\) on the same data \(X\).&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/defense.png&#34; alt=&#34;&#34; /&gt;
&lt;div class=&#34;caption&#34;&gt;TODO: add source&lt;/div&gt;&lt;/p&gt;

&lt;h3 id=&#34;results&#34;&gt;Results&lt;/h3&gt;

&lt;p&gt;This paper evaluated Resilience, Sensitivity and Robustness on 2 datasets: MNIST and CIFAR10&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/table.png&#34; alt=&#34;&#34; /&gt;
&lt;div class=&#34;caption&#34;&gt;
Resilience: success rate of adversarial crafting.
&lt;/div&gt;
&lt;div class=&#34;caption&#34;&gt;
  Sensitivity: amplitude of adversarial gradients.
&lt;/div&gt;
&lt;div class=&#34;caption&#34;&gt;
  Robustness: amount of perturbation required to achieve adversarial targets.
&lt;/div&gt;&lt;/p&gt;

&lt;p&gt;This paper also evaluated effect of Temperature on Adversarial Success
Success of adversarial samples when changing at most 112 features.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/res.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;These results seem promising, but subsequent work found that Defensive
Distillation is not actually robust to adversarial examples.&lt;/p&gt;

&lt;h3 id=&#34;why-distillation-seems-to-work&#34;&gt;Why Distillation Seems to Work&lt;/h3&gt;

&lt;p&gt;First some attacks use the gradient of the logits, where softmax do not work here. Second, big difference in relative impact of changes in input to the softmax layer. Third, training at temperature \(T\) effectively increases all inputs to the softmax layer by a factor of \(T\), for example, Undistilled logits with Mean of 5.8/std of 6.4 and Distilled logits (T=100) with Mean of 482/std of 457.&lt;/p&gt;

&lt;h3 id=&#34;breaking-distillation&#34;&gt;Breaking Distillation&lt;/h3&gt;

&lt;p&gt;Instead of using the gradient of the input to the softmax layer, the gradient of the output of the softmax layer was used.Due to the problem of vanishing gradients, artificially divide the inputs to the softmax by \(T\). This method achieves a successful misclassification rate of 96.4%.&lt;/p&gt;

&lt;h2 id=&#34;towards-evaluating-the-robustness-of-neural-networks&#34;&gt;Towards Evaluating the Robustness of Neural Networks&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Carlini, Nicholas, and David Wagner. &lt;em&gt;Towards evaluating the robustness of neural networks&lt;/em&gt;. IEEE Symposium on Security and Privacy (&amp;ldquo;Oakland&amp;rdquo;) 2017. [&lt;a href=&#34;https://arxiv.org/pdf/1608.04644.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1608.04644.pdf&#34;&gt;Carlini et al.&lt;/a&gt; proposed an optimization based attack to break the distillation defense mechanism developed by Papernot et al and other undistilled networks. This poses severe security problems to machine learning classifiers as the current defense strategy becomes vulnerable to this new type of strong white-box attacks. The attack is based on optimization approach, where the problem is formulated as&lt;/p&gt;

&lt;p&gt;$$ \text{minimize}~\mathcal{D}(x,x+\delta) + cf(x+\delta)\\ \text{such that:}~x+ \delta \in [0,1]^{n} $$&lt;/p&gt;

&lt;p&gt;where \(f(x+\delta) \leq 0\) iff the classification result of \((x+\delta)\) is in class \(t\) (i.e. \(C(x+\delta) = t)\). \(c\) is a coefficient controling the relative importance of misclassification and norm minimization. Intuitive understanding of the optimization problem above is, we want a smaller perturbation (unobservable by human) by minimizing the norm \(||\delta||\) and causing misclassification by minimizing the function \(f(x+\delta)\). The constraint of \(x+\delta \in [0,1]^{n}\) is because this attack is conducted in image domain and has to generate valid images whose pixel value is in \([0,1]\) range. In order to further avoid the constraint, the authors apply \(tanh\) function to change the variables. Specifically, \(\delta_i = \frac{1}{2}(tanh(w_i)+1)-x_i \) and we directly optimize the unconstrained variable \(w_i \) instead of \(\delta_i\). &lt;a href=&#34;https://arxiv.org/pdf/1412.6980.pdf&#34;&gt;ADAM&lt;/a&gt; is deployed to solve the above unconstrained optimization problem. A special note is the author implemented different \(p\)-norms \((p = 0, 2, \infty)\) under the optimization framework described above.&lt;/p&gt;

&lt;p&gt;The evaluation part is compared to other existing white-box attack methods. Specifically, &lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;Fast Gradient Sign (FGS)&lt;/a&gt;, &lt;a href=&#34;https://arxiv.org/pdf/1607.02533.pdf&#34;&gt;Iterative Fast Gradient Sign (IFGS)&lt;/a&gt; methods are suitable for \(L_\infty \)-norm, &lt;a href=&#34;https://arxiv.org/pdf/1511.04599.pdf&#34;&gt;Deepfool&lt;/a&gt; method is suitable for \(L_2\)-norm attack, &lt;a href=&#34;https://arxiv.org/pdf/1511.07528.pdf&#34;&gt;JSMA&lt;/a&gt; method is suitable for \(L_0\)-norm attack. The target model for the listed attacks are &lt;a href=&#34;https://arxiv.org/pdf/1511.04508.pdf&#34;&gt;distillation based network&lt;/a&gt; (defensitive strategy proposed for defending against FGS and IFGS methods) and normally trained neural networks. The final evaluation results demonstrate that the attack proposed in this work is far stronger than existing attack method in that it achieves highest attack success rate (this method achieves \(100\%\) success rate while other methods don&amp;rsquo;t) and lowest perturbation magnitude. Some sample images generated from this attack are shown below. This attack is also now broadly accepted as a standard baseline for evaluating newly proposed defense mechanisms.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/class3_img1.png&#34; alt=&#34;&#34; /&gt;
&lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://arxiv.org/pdf/1608.04644.pdf)&#34;&gt;&lt;em&gt;Towards evaluating the robustness of neural networks&lt;/em&gt;&lt;/a&gt; [1]
&lt;/div&gt;&lt;/p&gt;

&lt;h2 id=&#34;obfuscated-gradients&#34;&gt;Obfuscated Gradients&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Athalye, A., Carlini, N., &amp;amp; Wagner, D. &amp;ldquo;Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.&amp;rdquo; ArXiv e-prints, arXiv: 1802.00420. February 2018.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Optimization-based attacks such as the Basic Iterative Method, Projected Gradient Descent, and Carlini and Wagner’s attack are powerful threat to most defenses against adversarial examples in machine learning classifiers. New defenses have been proposed that appear to be resistant to optimization-based attacks, but Athalye, Carlini, and Wagner argue that these defenses are not as robust as they claim.&lt;/p&gt;

&lt;p&gt;Optimization-based attacks generate adversarial examples using gradients obtained through backpropagation. The authors identify obfuscated gradients as an element of many defenses that provides a facade of security by causing the attacker’s gradient descent to fail. Obfuscated gradients come in at least three forms: shattered gradients, stochastic gradients, and vanishing/exploding gradients.&lt;/p&gt;

&lt;p&gt;A defense uses &lt;strong&gt;shattered gradients&lt;/strong&gt; when it introduces non-differentiable operations, numeric instability, or otherwise causes the attacker’s gradient signal to be incorrect. A defense uses &lt;strong&gt;stochastic gradients&lt;/strong&gt; when the inputs to the classifier are randomized or a stochastic classifier is used, resulting in a different gradient each time it is evaluated. &lt;strong&gt;Vanishing/exploding gradients&lt;/strong&gt; are issues encountered in training some networks in which the gradient grows or shrinks exponentially during backpropagation. Some defenses involve multiple iterations of neural network evaluation. This can be viewed as evaluating one very deep neural network, which obfuscates the gradient signal by forcing the vanishing/exploding gradient problem.&lt;/p&gt;

&lt;h3 id=&#34;detecting-obfuscated-gradients&#34;&gt;Detecting Obfuscated Gradients&lt;/h3&gt;

&lt;p&gt;The authors propose a number of tests that might help detect when a defense relies on obfuscated gradients.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Iterative attacks should work better than single-step attacks, since iterative attacks are strictly stronger than single-step attacks.&lt;/li&gt;
&lt;li&gt;White-box attacks should perform better than black-box attacks, since the black-box threat model is a strict subset of the white-box threat model.&lt;/li&gt;
&lt;li&gt;Attacks with an unbounded distortion metric (e.g. \(L_2\) norm) should find adversarial examples with 100% success.&lt;/li&gt;
&lt;li&gt;Optimization-based attacks should perform better than brute-force sampling of nearby inputs (sampling within an \(\epsilon\)-ball).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These tests may not cover all cases of obfuscated gradients, but they indicate when intuitive properties start to break down. All defenses with obfuscated gradients discussed by the authors fail at least one test.&lt;/p&gt;

&lt;h3 id=&#34;attack-techniques&#34;&gt;Attack Techniques&lt;/h3&gt;

&lt;p&gt;The authors introduce attack techniques to address each of the three types of obfuscated gradients.&lt;/p&gt;

&lt;p&gt;Shattered gradients can be attacked using a technique called Backward Pass Differentiable Approximation. For a classifier \(f(\cdot)\), some defenses make gradients unavailable to attackers by incorporating a non-differentiable preprocessing step \(g(\cdot)\), making the secured classifier \(\hat{f}(\cdot) = f(g(\cdot))\). Backward Pass Differentiable Approximation estimates gradients by substituting a smooth, differentiable approximation of \(g\).&lt;/p&gt;

&lt;p&gt;Stochastic gradient-based defenses that randomly transform inputs can be defeated using the Expectation over Transformation technique to compute the gradient over the expected transformation of the input. If a stochastic classifier is used, the gradient can be similarly computed over the expectation of random parameters.&lt;/p&gt;

&lt;p&gt;Defenses that use vanishing/exploding gradients can be circumvented by reparameterizing the secured classifier and optimizing over a space where vanishing/exploding gradients do not occur.&lt;/p&gt;

&lt;p&gt;Seven of the eight defense techniques accepted to ICLR 2018 are based on a form of obfuscated gradients, and are therefore vulnerable to these attacks. The defense by Madry, et al., discussed below, was the only approach evaluated that doesn’t cause obfuscated gradients. Athalye, Carlini, and Wagner recommend that future defenses should be presented with specific, realistic threat models, testable claims with bounded test parameters, and evaluations against new defense-aware attacks.&lt;/p&gt;

&lt;h2 id=&#34;resistance-to-adversarial-attacks&#34;&gt;Resistance to Adversarial Attacks&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. &lt;em&gt;Towards Deep Learning Models Resistant to Adversarial Attacks&lt;/em&gt;.  ICLR 2018. [&lt;a href=&#34;https://arxiv.org/pdf/1706.06083.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1706.06083.pdf&#34;&gt;Madry et al.&lt;/a&gt; propose a general framework to study the defense of deep learning models against adversarial attacks.  The goal of their framework is to provide precise security guarantees about how a particular defense stacks up against entire classes of attacks.  Specifically, they offer the following saddle point problem:&lt;/p&gt;

&lt;p&gt;$$\underset{\theta}{\text{min}} \, \rho(\theta), \quad \text{where} \quad \rho(\theta) = \mathbb{E}_{(x, y) \sim \mathcal{D}} \bigg[ \underset{\delta \in \mathcal{S}}{\text{max}} \, L(\theta, x + \delta, y) \bigg] \, .$$&lt;/p&gt;

&lt;p&gt;Let’s break this down:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;\( \theta \) is the set of parameters of the model.  The attacker aims to exploit an existing model with a predetermined \( \theta \), and the defender can tune \( \theta \) as they wish.&lt;/li&gt;
&lt;li&gt;\( x \) and \( y \) are the example and label.&lt;/li&gt;
&lt;li&gt;\( L \) is the loss function.&lt;/li&gt;
&lt;li&gt;\( \mathbb{E} \) is a risk function.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is a classic saddle point problem, consisting of two connected parts: an inner maximization problem, and an outer minimization problem. An adversary seeks to find an example that maximizes the model’s loss (the inner problem), and the defender seeks to tune their model such that the potential loss is minimized (the outer problem).  Therefore, solving this problem (by minimizing rho) maximizes the robustness of a deep learning model.&lt;/p&gt;

&lt;p&gt;The inner loss function was modeled with a fast gradient sign method (FGSM) attack. The FGSM perturbed data was used for as training data on the defense side. The outer minimization function was a little more complicated. It’s not simple enough to train on FGSM adversaries. We want something that’s universally robust, so we use PGD, a first-order method to solve constrained optimization problems. The authors used PGD to find local maxima, since they are the areas of highest loss.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/Madry1.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;These graphs show how the loss scales as the number of PGD iterations increases.  The loss grows and plateaus as the number of iterations grows (as expected for any model).  However, the adversarially-trained models display much less loss than the naturally-trained models, indicating that Madry et. al.’s defense works well against their attacks.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://secml.github.io/images/class3/Madry2.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;

&lt;p&gt;These graphs show the distributions of loss values, for various starting points.  The loss values are obtained by randomly sampling points within an \(L\infty\)-ball centered at the starting point, then using PGD to find the local maxima near those random points.  Although the locations of the points are widely distributed, the loss values are all clustered together.  The authors claim that these graphs display the universality of PGD: any local maxima with significantly higher losses would likely be infeasible for any other first-order method to find.&lt;/p&gt;

&lt;p&gt;— Team Gibbon: &lt;br /&gt;
Austin Chen, Jin Ding, Ethan Lowman, Aditi Narvekar, Suya&lt;/p&gt;

&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1608.04644.pdf&#34;&gt;[1]&lt;/a&gt; Carlini N, Wagner D. Towards evaluating the robustness of neural networks. InSecurity and Privacy (SP), 2017 IEEE Symposium on 2017 May 22 (pp. 39-57). IEEE.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1412.6980.pdf&#34;&gt;[2]&lt;/a&gt; Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014 Dec 22.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;[3]&lt;/a&gt; Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014 Dec 20.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1607.02533.pdf&#34;&gt;[4]&lt;/a&gt; Kurakin A, Goodfellow I, Bengio S. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533. 2016 Jul 8.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1511.04599.pdf&#34;&gt;[5]&lt;/a&gt; Moosavi Dezfooli SM, Fawzi A, Frossard P. Deepfool: a simple and accurate method to fool deep neural networks. InProceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 (No. EPFL-CONF-218057).&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1511.07528.pdf&#34;&gt;[6]&lt;/a&gt; Papernot N, McDaniel P, Jha S, Fredrikson M, Celik ZB, Swami A. The limitations of deep learning in adversarial settings. InSecurity and Privacy (EuroS&amp;amp;P), 2016 IEEE European Symposium on 2016 Mar 21 (pp. 372-387). IEEE.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1511.04508.pdf&#34;&gt;[7]&lt;/a&gt; Papernot N, McDaniel P, Wu X, Jha S, Swami A. Distillation as a defense to adversarial perturbations against deep neural networks. InSecurity and Privacy (SP), 2016 IEEE Symposium on 2016 May 22 (pp. 582-597). IEEE.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cs.cornell.edu/~shmat/shmat_oak17.pdf&#34;&gt;[8]&lt;/a&gt; A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, &amp;ldquo;Towards Deep Learning Models Resistant to Adversarial Attacks&amp;rdquo;.  February 2018.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Class 2: Privacy in Machine Learning</title>
      <link>https://secml.github.io/class2/</link>
      <pubDate>Fri, 02 Feb 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class2/</guid>
      <description>

&lt;p&gt;In today’s post we introduce some key concepts crucial to understanding the current state of privacy in machine learning. In a time where novel machine learning applications are seemingly announced weekly, privacy is becoming more relevant as learning algorithms play varied and sometimes critical roles in our lives. We introduce differential privacy and common ‘solutions’ that fail to protect individual privacy, explore membership inference attacks on blackbox machine learning models, and discuss a case study involving privacy in the field of pharmacogenetics, where machine learning models are used to guide patient treatment.&lt;/p&gt;

&lt;h2 id=&#34;membership-inference-attacks&#34;&gt;Membership inference attacks&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. &lt;em&gt;Membership Inference Attacks Against Machine Learning Models&lt;/em&gt;. IEEE Symposium on Security and Privacy (&amp;ldquo;Oakland&amp;rdquo;) 2017. [&lt;a href=&#34;https://www.cs.cornell.edu/~shmat/shmat_oak17.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cs.cornell.edu/~shmat/shmat_oak17.pdf&#34;&gt;Shokri et al.&lt;/a&gt; attempt to attack black box machine learning models based on subtle data leaks based on the outputs. If the membership of a datapoint can be identified in the training set of a black box machine, it poses a significant privacy risk to the data of users of machine learning services. This is especially important for machine learning services such as Google Prediction and Amazon ML, as an inability to guarantee the privacy of training data would preclude a significant number of customers from using their services.&lt;/p&gt;

&lt;p&gt;The attack is based on the idea that there are noticeable patterns in outputs when the input is given to the model as training data. While no human would be able to pick up on a pattern of that nature, a machine learning model with a theoretically limitless data source has that power. As the purpose of machine learning is to notice and evaluate patterns beyond human recognition, using a machine learning model to attack black box machine learning models makes sense.&lt;/p&gt;

&lt;p&gt;For a machine learning model to attack the black box, it first needs to train against other models where it can verify its own accuracy. For this, similar models need to be created for evaluation. These models are called shadow models, and need to be trained on similar sample data to the original model. To generate training data, these models use hill climbing methods on the black box to seek high confidence areas. By then sampling data using a simple distribution about those points, somewhat accurate training sets can be created. Ideally, these sets will be roughly the same size and distribution of the target model.&lt;/p&gt;

&lt;p&gt;The attack model is then trained on a set of shadow models, using classifications based on whether or not the datapoint was in the training set of the shadow model. This theoretically works based on the attack model being able to locate areas in which the curves are overfitted in the shadow models and translate that into membership. Ultimately, the study produced results that ranged from 70% to 95% precision overall based on the target model. The model gave nearly no false negatives.&lt;/p&gt;

&lt;p&gt;To limit the power of these attacks, more regularization can be used to minimize overfitting, as the overfitting patterns become less apparent as the curve fitting is minimized. However, differential privacy is the more general class of solutions that is the defining foundation of the current state of the field of privacy in machine learning.&lt;/p&gt;

&lt;h2 id=&#34;differential-privacy&#34;&gt;Differential privacy&lt;/h2&gt;

&lt;p&gt;With the explosion of data collection from various social platforms and the increasing usage of machine learning on personalized applications like personalized medication, medical diagnostics, genomics and face recognition, the collection of sensitive human data is inevitable. When dealing with personal information, it is paramount to preserve the privacy of any individual participating in the data set. In a word, it is necessary to ensure that an adversary does not learn about the presence or absence of any individual’s information from looking at the synopsis of the data set. This goal is the central motivation the concept of differential privacy.&lt;/p&gt;

&lt;p&gt;To fully appreciate differential privacy, let us discuss some &lt;a href=&#34;http://www.lrdc.pitt.edu/schunn/cdi2009/presentations/Dwork.pdf&#34;&gt;alternative solutions&lt;/a&gt; to preserving individual privacy in a data set identify how each one fails in achieving the goal of privacy.&lt;/p&gt;

&lt;h4 id=&#34;failed-alternatives&#34;&gt;Failed alternatives&lt;/h4&gt;

&lt;h6 id=&#34;1-allowing-only-group-queries&#34;&gt;1. Allowing only group queries&lt;/h6&gt;

&lt;p&gt;It seems intuitive that individual data privacy can be preserved by restricting a user to only group queries. For example, a group query can be: “How many students in this class earned an A?”. But this blatantly violates the privacy if the user also queries “How many students in this class apart from Rick earned an A?”, where the end user can know if Rick received an A or not based on the answer to the two queries.&lt;/p&gt;

&lt;h6 id=&#34;2-add-random-noise-to-the-query-result&#34;&gt;2. Add random noise to the query result&lt;/h6&gt;

&lt;p&gt;Adding random noise to the query result tends to obfuscate the relation between the input and the output, but an adversary can determine the noise distribution by repeated querying. Hence, this approach also fails.&lt;/p&gt;

&lt;h6 id=&#34;3-deterministic-perturbation-in-answering&#34;&gt;3. Deterministic perturbation in answering&lt;/h6&gt;

&lt;p&gt;We can deterministically map every output to another value in the vector space to obfuscate the input-output relation. But this becomes far too computationally complex in a high-dimensional regime.&lt;/p&gt;

&lt;h6 id=&#34;4-withholding-sensitive-information&#34;&gt;4. Withholding sensitive information&lt;/h6&gt;

&lt;p&gt;An easy, straightforward notion is to simply withhold any sensitive data before executing any query; however, this very action reveals information about the presence or absence of sensitive data.&lt;/p&gt;

&lt;h6 id=&#34;5-make-output-independent-of-any-individual-record&#34;&gt;5. Make output independent of any individual record&lt;/h6&gt;

&lt;p&gt;This approach certainly does not reveal the presence or absence of an individual in the data set. But this method is not of any practical use since, by mathematical induction, the output does not depend on the data set itself.&lt;/p&gt;

&lt;h6 id=&#34;6-k-anonymity&#34;&gt;6. K-anonymity&lt;/h6&gt;

&lt;p&gt;Another way to achieve obfuscation of a record is to group it with \(k\) other records for each attribute. Unfortunately, this method also fails in high-dimensional cases, where it has been shown to be feasible to still distinguish an individual record.&lt;/p&gt;

&lt;h6 id=&#34;7-semantic-security&#34;&gt;7. Semantic Security&lt;/h6&gt;

&lt;p&gt;Semantic security is a strict notion in cryptography which ensures that the advantage of an adversary with an auxiliary information should be cryptographically small with or without the access to the output (ciphertext). In our query model, this definition is too strong and does not allow learning any useful information from the data set.&lt;/p&gt;

&lt;h4 id=&#34;defining-differential-privacy&#34;&gt;Defining differential privacy&lt;/h4&gt;

&lt;p&gt;The &lt;a href=&#34;https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf&#34;&gt;formal definition&lt;/a&gt; of differential privacy is given as:&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;A randomized computation \(M\) satisfies \(\epsilon\)-differential privacy if for any adjacent data sets \(x\) and \(x’\), and any subset \(C\) of possible outcomes \(Range(M)\),&lt;/p&gt;

&lt;p&gt;$$Pr[M(x) \in C] \leq \exp(\epsilon) \times Pr[M(x&amp;rsquo;) \in C]$$&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;In other words, this means that the chance that an event occurs &lt;em&gt;with your data&lt;/em&gt; and the chance it would occur &lt;em&gt;without your data&lt;/em&gt; is closely bounded by a privacy budget \(\epsilon\). The figure below depicts the equation where the two lines denote the probability distribution over \(x\) and \(x’\).&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://raw.githubusercontent.com/secML/secML.github.io/master/src/content/images/img1.png&#34; alt=&#34;&#34; title=&#34;Response probability distribution&#34; /&gt;
   &lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(http://www.lrdc.pitt.edu/schunn/cdi2009/presentations/Dwork.pdf)&#34;&gt;&lt;em&gt;Differential Privacy and Pan-Private Algorithms&lt;/em&gt;&lt;/a&gt; [6]
   &lt;/div&gt;&lt;/p&gt;

&lt;p&gt;There is another relaxed version of the above definition of differential privacy, called \((\epsilon,\delta)\)-differential privacy, &lt;a href=&#34;https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf&#34;&gt;defined&lt;/a&gt; as below:&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;A randomized computation \(M\) satisfies \((\epsilon,\delta)\)-differential privacy if for any adjacent data sets \(x\) and \(x’\), and any subset \(C\) of possible outcomes \(Range(M)\),&lt;/p&gt;

&lt;p&gt;$$Pr[M(x) \in C] \leq \exp(\epsilon) \times Pr[M(x&amp;rsquo;) \in C] + \delta$$&lt;/p&gt;

&lt;hr /&gt;

&lt;p&gt;The above definition basically implies that with probability \((1-\delta)\), \(M\) preserves \(\epsilon\)-differential privacy.&lt;/p&gt;

&lt;h4 id=&#34;properties-of-differential-privacy&#34;&gt;Properties of differential privacy&lt;/h4&gt;

&lt;p&gt;&lt;strong&gt;Group Privacy&lt;/strong&gt;: An \(\epsilon\)-differentially private algorithm can be extended to provide group privacy for a group size of \(k\) by scaling the privacy budget to \(k\epsilon\).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Composition&lt;/strong&gt;: Composition of \(k \epsilon\)-differential privacy and \((\epsilon,\delta)\)-differential privacy methods leads to \(k \epsilon\)-differential privacy and \((k \epsilon, k \delta)\)-differential privacy methods respectively.&lt;/p&gt;

&lt;h4 id=&#34;differential-privacy-with-laplace-noise&#34;&gt;Differential privacy with Laplace noise&lt;/h4&gt;

&lt;p&gt;To ensure \(\epsilon\)-differential privacy for a function \(f(x)\), we can add &lt;a href=&#34;https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf&#34;&gt;Laplace noise&lt;/a&gt; to it:&lt;/p&gt;

&lt;p&gt;$$ f(x) + \text{Laplace}(0, \sigma)^d $$&lt;/p&gt;

&lt;p&gt;where \(\Delta f = \text{max} || f(x)-f(x&amp;rsquo;) ||_1 \)  and \(\sigma \geq \frac{\Delta f}{\epsilon}\).&lt;/p&gt;

&lt;p&gt;Let us consider a working example. Consider that there are two adjacent data sets \(D\) and \(D&amp;rsquo;\) consisting of scores of students of a class, such that \(D\) and \(D’\) differ by only one student’s record.&lt;/p&gt;

&lt;p&gt;Consider the function \(f\) that computes the class average. If, say, the minimum and maximum attainable scores are \(0\) and \(100\) respectively, then the sensitivity of \(f\) is \( \frac{100-0}{4} = 25\).&lt;/p&gt;

&lt;p&gt;Thus, to release the class average of \(D\) is given by
\(\frac{65+83+77+56}{4}\) with \(\epsilon\)-differential privacy,
we would add the result of sampling \(\text{Laplace}(0,
\frac{25}{\epsilon})\) to the computed value. This preserves the
privacy of the student differing in \(D\) and \(D’\).&lt;/p&gt;

&lt;h2 id=&#34;privacy-in-pharmacogenetics&#34;&gt;Privacy in Pharmacogenetics&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Matthew Fredrikson, Eric Lantz, Eric and Jha, Somesh and Lin, Simon and Page, David and Ristenpart, Thomas
&lt;em&gt;Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing&lt;/em&gt;. USENIX Security Symposium 2014. [&lt;a href=&#34;https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-fredrikson-privacy.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;In their case study, &lt;a href=&#34;https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-fredrikson-privacy.pdf&#34;&gt;Fredrikson et al.&lt;/a&gt;
analyze personalized Warfarin doses, a widely used anticoagulant. Finding the correct dose of Warfarin is vitally important as both low and high doses may result in death of the patient &amp;ndash; literally, a matter of life and death.&lt;/p&gt;

&lt;p&gt;This experiment was done on the available IWPC dataset which contains and uses demographic information &lt;em&gt;(age, height, weight, race)&lt;/em&gt;, genetic markers &lt;em&gt;(CYP2C9/VKORC1)&lt;/em&gt; and clinical histories of people around the world to predict their appropriate Warfarin dose. The IWPC has found that linear regression is the best learning model to fit the data and has made the model available for physicians&amp;rsquo; use in calculating the initial dose of the Warfarin.&lt;/p&gt;

&lt;p&gt;Let’s highlight how an adversary can use model inversion attack to violate the genomic privacy of a patient. We first initially assume that an adversary has black-box access to the trained model. As it turns out, due to the highly linear nature of the data, by running the model backwards, the sensitive genotype can be predicted given the stable dose of Warfarin, some basic demographic facts regarding the patient, and the marginal priors of the patient distribution.&lt;/p&gt;

&lt;p&gt;How does the attack work? First, the adversary computes all the values of the missing variables that could potentially agree with the given information. Then, the adversary runs the model forward for each hypothetical patient in the dataset to predict the stable Warfarin doses. Finally, the adversary performs a likelihood computation to find out which configuration of the missing variables are most probable. Given the information and model inversion setting, this algorithm is optimal, as it minimizes the adversaries misprediction rate. With respect to the baseline, the accuracy of the model inversion attack is only 5% lower in comparison to ideal prediction.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://raw.githubusercontent.com/secML/secML.github.io/master/src/content/images/img2.png&#34; alt=&#34;&#34; title=&#34;Model inversion attack&#34; /&gt;
   &lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;https://www.usenix.org/sites/default/files/conference/protected-files/sec14_slides_fredrikson.pdf&#34;&gt;Privacy in Pharmacogenetics presentation&lt;/a&gt; [2]
   &lt;/div&gt;&lt;/p&gt;

&lt;p&gt;As seen in this image, the attacker computes the values of missing variables, then predicts the Warfarin doses; and finally, using this information, finds the most likely configuration.&lt;/p&gt;

&lt;p&gt;Adding noise using differential privacy strategy can be a countermeasure against the model inversion attack. But using differential privacy decreases the utility of the trained model. Based on their simulated trials, authors claim that there is no such privacy budget that can prevent model inversion, without introducing  risk of overfixed dosing.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://raw.githubusercontent.com/secML/secML.github.io/master/src/content/images/img3.png&#34; alt=&#34;&#34; title=&#34;Privacy budget plot&#34; /&gt;
   &lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;https://www.usenix.org/sites/default/files/conference/protected-files/sec14_slides_fredrikson.pdf&#34;&gt;Privacy in Pharmacogenetics presentation&lt;/a&gt; [2]
   &lt;/div&gt;&lt;/p&gt;

&lt;h2 id=&#34;the-netflix-prize-semi-supervised-deep-learning-from-private-training-data&#34;&gt;The Netflix Prize: Semi-supervised deep learning from private training data&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;Frank McSherry and Ilya Mironov, &lt;em&gt;Differentially Private Recommender Systems: Building Privacy into the Netflix Prize Contenders&lt;/em&gt;. KDD 2009. [&lt;a href=&#34;https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf&#34;&gt;PDF&lt;/a&gt;]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Traditional recommender systems are usually not designed with an emphasis on privacy, which can be detrimental when the adversary are able to create multiple accounts to affect the recommendation at will. &lt;a href=&#34;https://www.cs.utexas.edu/~shmat/shmat_oak08netflix.pdf&#34;&gt;Researchers&lt;/a&gt; have been successful in developing powerful attack linking records in the Netflix Prize data set with other public user data. The implication is that sensitive information as the input of a recommender system can be linked and made inference about. &lt;a href=&#34;https://www.cs.utexas.edu/~shmat/shmat_oak11ymal.pdf&#34;&gt;Similar practical examples&lt;/a&gt; include making inference about purchase history through Amazon’s recommendations.&lt;/p&gt;

&lt;p&gt;McSherry and Mironov’s &lt;a href=&#34;https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf&#34;&gt;paper&lt;/a&gt; adapted the leading recommendation algorithms (factor models and neighborhood models) to the differential privacy framework in order to develop a recommender system based on the Netflix data set while providing privacy guarantees. As mentioned in the above section, differential privacy enables privacy preserving computation by substantially precluding inference from the output data, unlike the prior efforts focusing mainly on cryptographic solutions that limit access to data of users. By doing so, it is conceivable that more uncertainty/noise is added to computation.&lt;/p&gt;

&lt;p&gt;One interesting aspect this paper dealt with is the privacy vs. accuracy tradeoff. Evaluation of their approach was applied to the Netflix Prize data set which bears extremely high dimensionality. The root mean squared error (RMSE) was used as the metric for accuracy. The findings are as the quality of privacy increases, the accuracy of the recommendation drops.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://raw.githubusercontent.com/secML/secML.github.io/master/src/content/images/img4.png&#34; alt=&#34;&#34; title=&#34;RMSE for four algorithms as a function of θ \\(\propto\\\) 1/σ on the Netflix Prize set.&#34; /&gt;
   &lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf)&#34;&gt;&lt;em&gt;Differentially Private Recommender Systems&lt;/em&gt;&lt;/a&gt; [3]
   &lt;/div&gt;&lt;/p&gt;

&lt;p&gt;Privacy vs. accuracy over time was also studied to explore how the loss of accuracy due to privacy-preserving decreased as more data become available for a fixed value of the privacy parameter. The results are summarized in the following figure.&lt;/p&gt;

&lt;p&gt;&lt;img src=&#34;https://raw.githubusercontent.com/secML/secML.github.io/master/src/content/images/img5.png&#34; alt=&#34;&#34; title=&#34;Left scale—accuracy loss, right scale—the
number of records. The \\(x\\\)-axis is the number of days
elapsed since 7/1/2000.&#34; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;div class=&#34;caption&#34;&gt;
Source: &lt;a href=&#34;(https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf)&#34;&gt;&lt;em&gt;Differentially Private Recommender Systems&lt;/em&gt;&lt;/a&gt; [3]
   &lt;/div&gt;&lt;/p&gt;

&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;

&lt;p&gt;We have introduced differential privacy and formally defined what is meant by preservation of \(\epsilon\)-differential privacy; discussed common solutions and how they fail; explained the properties of a differentially private algorithm, and how adding Laplace Noise enables \(\epsilon\)-differential private functions. We also discussed how in Membership Inference Attacks Against Machine Learning Models, Shokri et al. discuss attacking a black box model by training on a set of shadow models. Finally,  we discussed how in Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing Fredrikson et al. demonstrate how a model inversion attack can be employed to infer patient genotype information.&lt;/p&gt;

&lt;p&gt;— Team Nematode: &lt;br /&gt;
Bargav Jayaraman, Guy &amp;ldquo;Jack&amp;rdquo; Verrier, Joshua Holtzman, Max Naylor, Nan Yang, Tanmoy Sen&lt;/p&gt;

&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cs.cornell.edu/~shmat/shmat_oak17.pdf&#34;&gt;[1]&lt;/a&gt; R. Shokri, M. Stronati, C. Song, V. Shmatikov, &amp;ldquo;Membership Inference Attacks Against Machine Learning Models.&amp;rdquo; May 2017.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-fredrikson-privacy.pdf&#34;&gt;[2]&lt;/a&gt; M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, T. Ristenpart, &amp;ldquo;Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.&amp;rdquo; August 2014.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.microsoft.com/en-us/research/wp-content/uploads/2009/06/NetflixPrivacy.pdf&#34;&gt;[3]&lt;/a&gt; F. McSherry, I. Mironov, &amp;ldquo;Differentially Private Recommender Systems: Building Privacy into the Netflix Prize Contenders.&amp;rdquo; June 2009.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cs.utexas.edu/~shmat/shmat_oak08netflix.pdf&#34;&gt;[4]&lt;/a&gt; A. Narayanan, V. Shmatikov, &amp;ldquo;Robust De-anonymization of Large Sparse Datasets.&amp;rdquo; May 2008.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.cs.utexas.edu/~shmat/shmat_oak11ymal.pdf&#34;&gt;[5]&lt;/a&gt; J. A. Calandrino, A. Kilzer, A. Narayanan, E. W. Felten, V. Shmatikov, &amp;ldquo;&amp;lsquo;You Might Also Like&amp;rsquo;: Privacy Risks of Collaborative Filtering.&amp;rdquo; May 2011.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;http://www.lrdc.pitt.edu/schunn/cdi2009/presentations/Dwork.pdf&#34;&gt;[6]&lt;/a&gt; C. Dwork, &amp;ldquo;Differential Privacy and Pan-Private Algorithms.&amp;rdquo; August 2016.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://www.youtube.com/watch?v=OfWj89oRD7g&#34;&gt;[7]&lt;/a&gt; C. Dwork, &amp;ldquo;Differential Privacy - Lecture 1&amp;rdquo;. August 2016.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Class 1: Intro to Adversarial Machine Learning</title>
      <link>https://secml.github.io/class1/</link>
      <pubDate>Fri, 26 Jan 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/class1/</guid>
      <description>

&lt;h2 id=&#34;machine-learning-background&#34;&gt;Machine Learning Background&lt;/h2&gt;

&lt;!-- - Other ML algorithms use linear decision boundaries (SVM, LR, ...)
- Deep learning uses linear units with nonlinear composition
    - More susceptible to attacks
- Define terms like gradient and loss --&gt;

&lt;p&gt;In supervised Machine Learning, we train models with training data
along with the label associated with it. We extract features from each
sample, and use an algorithm to train a model where the inputs are
those features and the output is the label.&lt;/p&gt;

&lt;p&gt;For classifying the testing data, the classifier uses decision
boundary to separate points of the data belonging to each class. In a
statistical-classification problem with two classes, a decision
boundary partitions all the underlying vector space into two separate
classes. A support vector machine (SVM) is a linear classifier which
constructs a boundary by focusing two closest points from different
classes and it finds the line that is equidistant to these two points.&lt;/p&gt;

&lt;h4 id=&#34;loss-functions&#34;&gt;Loss Functions&lt;/h4&gt;

&lt;p&gt;A loss function define that how good a given model is at making
predictions for a given scenario. It has its own curve and its own
gradients. The slope of the curve indicates the appropriate way of
updating the parameters to make the model more accurate in case of
prediction. A frequently used loss function is the 0-1 loss function.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/01_loss_function.png&#34; width=&#34;150&#34; &gt;
&lt;/p&gt;

&lt;p&gt;where \(I\) is the indicator notation. Hinge loss function is also popular in machine learning field. It provides a relatively tight, convex upper bound on the 0-1 indicator function.
s
&lt;!-- &lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/hinge_function.png&#34; width=&#34;150&#34; &gt;
&lt;/p&gt; --&gt;&lt;/p&gt;

&lt;h4 id=&#34;gradient-descent&#34;&gt;Gradient Descent&lt;/h4&gt;

&lt;p&gt;Gradient Descent is an optimization algorithm which is used to minimize cost function by iteratively moving the direction of the steepest descent. In machine learning, we use gradient descent to update the parameters (coefficients in Linear Regression and weight in neural networks) of our model.&lt;/p&gt;

&lt;p&gt;&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/gradient_descent_graph.png&#34; width=&#34;300&#34; &gt;
&lt;br&gt;
Image credit: &lt;a href=&#34;https://sebastianraschka.com/faq/docs/closed-form-vs-gd.html&#34;&gt;Sebastian Raschka&lt;/a&gt;
   &lt;/p&gt;&lt;/p&gt;

&lt;p&gt;In the figure, weight is slowly decreasing from the initial stage. \(J_{min}\) is the minimum value of the cost function which can be obtained by optimizing the algorithm.&lt;/p&gt;

&lt;p&gt;Recently, there have been many successful applications of Deep Neural
Networks (DNN) in the fields of information retrieval, computer
vision, and speech recognition. Despite their potential, deep neural
architectures, like all other machine learning approaches, are
vulnerable to what are known as adversarial samples. These systems can
be fooled by targeted manipulations which slightly modifying a real
example to trick the model into “believing” that this modified sample
belongs to an incorrect class with high confidence. To defend against
these adversarial attacks, many researchers are exploring several
directions including data augmentation, increasing model complexity,
retraining, pre-processing, etc. Their main goal is to protect the
model against adversarial manipulations by the attackers.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;applications-and-vulnerabilities&#34;&gt;Applications and Vulnerabilities&lt;/h2&gt;

&lt;p&gt;To get a better idea of what security and privacy mean in a machine learning context, we highlight a few
applications and examine the possible consequences of vulnerabilities in these domains.&lt;/p&gt;

&lt;p&gt;One of the most commonly-discussed applications is in self-driving
cars, which use machine learning algorithms for image recognition
tasks, such as identifying traffic signs and road hazards.  If an
attacker were to cause an erroneous classification, e.g. mistaking a
stop sign for a speed limit sign or missing a pedestrian in a
crosswalk, both property and human lives would be in
jeopardy. (Although luckily self-driving cars use a variety of other
techniques to avoid collisions, including detailed maps and LIDAR, so
are likely to not be directly vulnerable to image mis-classification
attacks.)&lt;/p&gt;

&lt;p&gt;Machine learning has also found a place in the medical world, being used to identify patterns and trends in patient histories that can be used for diagnoses, to recognize abnormalities in medical imaging, and even to evaluate the efficiency of a hospital&amp;rsquo;s workflow.  The use of machine learning in healthcare introduces a new concern - how can patient data be effectively utilized for beneficial purposes, like predicting appropriate drug dosages, without compromising patient privacy?&lt;/p&gt;

&lt;p&gt;There are numerous other uses of ML today (targeted advertising, fraud detection, malware detection, etc.) and each carries particular security and privacy concerns.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;adversarial-examples&#34;&gt;Adversarial Examples&lt;/h2&gt;

&lt;p&gt;In machine learning, an adversarial example is an input that has been manipulated so that the model returns a different, incorrect output.  A classic adversarial example for image classification is from a paper by &lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;Goodfellow et al.&lt;/a&gt;, which features a photo of a panda that was originally classified correctly, but is misclassified when carefully-crafted noise is added.  The model classifies the new image as a gibbon, even though human eyes can easily see it is a panda.&lt;/p&gt;

&lt;p&gt;&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/panda.png&#34; width=&#34;600&#34; &gt;
&lt;br&gt; Source: &lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;Goodfellow Paper&lt;/a&gt;
   &lt;/p&gt;&lt;/p&gt;

&lt;p&gt;As outlined in &lt;a href=&#34;https://arxiv.org/pdf/1712.03141.pdf&#34;&gt;Biggio et al.&amp;rsquo;s&lt;/a&gt; paper on the history of adversarial machine learning, adversarial examples were first described in 2004 in the context of email spam filters. At this time, &lt;a href=&#34;https://homes.cs.washington.edu/~pedrod/papers/kdd04.pdf&#34;&gt;Dalvi et al.&lt;/a&gt; and &lt;a href=&#34;https://ix.cs.uoregon.edu/~lowd/kdd05lowd.pdf&#34;&gt;Lowd and Meek&lt;/a&gt; realized that slight modifications to spam emails allowed them to pass through the filters, without greatly affecting the content of the message. From this point, the field expanded to study the potential for adversarial examples in other realms, like machine-learning-based image and malware classification.  The increased usage of deep learning techniques for object recognition led to a surge in interest around 2014, when Szegedy et al. showed that deep convolutional neural networks were also susceptible to adversarial examples.  Since then, interest in this field has only continued to increase, with more and more papers published each year.&lt;/p&gt;

&lt;h3 id=&#34;classifying-attacks&#34;&gt;Classifying Attacks&lt;/h3&gt;

&lt;p&gt;Attacks can be categorized by many different characteristics, but are often referred to in terms of the attack method, the goal of the attack, and the adversary&amp;rsquo;s level of knowledge, as detailed in &lt;a href=&#34;https://arxiv.org/pdf/1611.03814.pdf&#34;&gt;Papernot et al.&lt;/a&gt;.&lt;/p&gt;

&lt;h4 id=&#34;attack-method&#34;&gt;Attack Method&lt;/h4&gt;

&lt;p&gt;A poisoning attack is when the adversary adds carefully-crafted samples into the training data, thereby disrupting the learning process and manipulating the final trained model.  An evasion attack occurs when a sample originally correctly classified into class A is manipulated and fed back into the model, now receiving an output that is &lt;em&gt;not&lt;/em&gt; class A.  A key difference between these two attacks is where each occurs in the ML pipeline: poisoning attacks occur before the training stage and evasion attacks occuring after training, during the testing stage.&lt;/p&gt;

&lt;p&gt;&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/ml_pipeline.jpg&#34; width=&#34;650&#34; &gt;
&lt;br&gt; Source: &lt;a href=&#34;https://docs.google.com/presentation/d/1dFRjRfCIz1TChZYfIY_FLYiSk7nNmdBY2luJ8T_TCAE/edit?usp=sharing&#34;&gt;Presentation Slides&lt;/a&gt;
   &lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;goal-of-attack&#34;&gt;Goal of Attack&lt;/h4&gt;

&lt;p&gt;The goal of the adversary is another way to categorize attacks, which
in this case means whether or not there was a specific goal
classification or not.  &lt;em&gt;Error-generic&lt;/em&gt; (also called &lt;em&gt;untargeted&lt;/em&gt;)
attacks aim to find a sample close to a given seed that is
misclassified, but do not have a specific target output
class. &lt;em&gt;Error-specific&lt;/em&gt; (&lt;em&gt;targeted&lt;/em&gt;) attacks deliberately change the
seed sample&amp;rsquo;s classification from the original class A to a chosen
class B.&lt;/p&gt;

&lt;h4 id=&#34;adversary-knowledge&#34;&gt;Adversary Knowledge&lt;/h4&gt;

&lt;p&gt;The level of knowledge the adversary possesses generally falls into one of three loosely defined categories: black box, grey box, and white box attacks.  Black box attacks assume the lowest level of adversary knowledge, generally only allowing the final classification to be known without any finer details about the model.  On the other end of the spectrum, white box attacks assume that the attacker knows everything about the model, including the specific algorithm used, the feature set, the training data, etc. An attack by an adversary who knows more about the model than a black box but less than a white box is called a grey box attack, although the exact definition tends to vary.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/boxes.png&#34; width=&#34;600&#34; &gt;
&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;modern-machine-learning-as-a-potemkin-village&#34;&gt;Modern Machine Learning as a Potemkin village&lt;/h2&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/potemkin.jpeg&#34; width=&#34;350&#34; &gt;
&lt;/p&gt;

&lt;p&gt;The term &amp;ldquo;Potemkin village&amp;rdquo; describes a difference between appeance and reality. In a &lt;a href=&#34;https://arxiv.org/abs/1412.6572&#34;&gt;paper&lt;/a&gt; by Goodfellow, Shlens, and Szegdy, modern machine learning methods are described as building &amp;ldquo;a Potemkin village that works well on naturally occuring data, but is exposed as a fake when one visits points in space that do not have high probability in the data distribution.&amp;rdquo; The apparent fragility of deep neural networks has been particuarly exposed, as seen in the panda picture above.&lt;/p&gt;

&lt;p&gt;The two-faced nature of machine learning models to be both fragile and robout from two different angles leads to concerns in the security of machine learning algorithms. High confidence in classifying adversarial examples poses a threat to the integrity of the model&amp;rsquo;s output.&lt;/p&gt;

&lt;h4 id=&#34;fast-gradient-sign-method-fgsm&#34;&gt;Fast gradient sign method (FGSM)&lt;/h4&gt;

&lt;p&gt;Also the &lt;a href=&#34;https://arxiv.org/abs/1412.6572&#34;&gt;paper&lt;/a&gt; mentioned a method for generating adversarial examples. Fast gradient sign method maximizes the error between the ground truth classification of the sample and minimizes the error to the target classification. The method is effecitve is generating the best pixels to change to achieve a higher loss, essentially acting as a reverse optimization method. FGSM simply traverses the loss curve by moving in the opposite direction of the gradient loss.&lt;/p&gt;

&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/formula.png&#34; width=&#34;200&#34; &gt;
&lt;/p&gt;

&lt;p&gt;As we discussed, the sign method in the FGSM is an approximation to the actual gradient. Although the most accurate method would use the true gradient, the sign of the gradient is taken for the sake efficiency: since \(\epsilon\) reflect the adversarial strength (the maximum size perturbation allowed), this represents taking the largest step possible in each dimension in the direction given by the gradient. (There are many variations on FGSM that we&amp;rsquo;ll discuss in later posts, including iterative versions where a sequence of smaller steps is used.) FGSM showcases the fragility of machine learning models especially through visualizations such as the panda example.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;adversarial-training&#34;&gt;Adversarial training&lt;/h2&gt;

&lt;h4 id=&#34;effects-of-nonlinearities&#34;&gt;Effects of nonlinearities&lt;/h4&gt;

&lt;p&gt;One criticism of deep neural networks is that their nonlinearity creates vulnerabilities that can be exploited by adversarial examples.  As shown below, the mode function cuts through the data points linearly, while the function drawn by the neural network conforms more tightly to the training data points.  This overfitting to training data creates pockets in which the class chosen by the function does not match the ground truth class value for a given data point.&lt;/p&gt;

&lt;p&gt;&lt;p align=&#34;center&#34;&gt;
&lt;img src=&#34;https://secml.github.io/images/capacity.png&#34; width=&#34;650&#34; &gt;
&lt;br&gt;Image credit: &lt;a href=&#34;https://www.papernot.fr/files/16-mcdaniel-sp-machine-learning-in-adversarial-settings.pdf&#34;&gt;McDaniel, Papernot, and Celik, &lt;em&gt;IEEE Security &amp;amp; Privacy Magazine&lt;/em&gt;&lt;/a&gt;
   &lt;/p&gt;&lt;/p&gt;

&lt;h4 id=&#34;solutions&#34;&gt;Solutions&lt;/h4&gt;

&lt;p&gt;As a possible solution, &lt;a href=&#34;https://arxiv.org/pdf/1312.6199.pdf&#34;&gt;Szegedy et al.&lt;/a&gt; proposed incorporating adversarial examples into the training data.  Using various techniques to find the vulnerable pockets in the model, the authors were able to generate examples that would fall into those adversarial region.  Adding those samples to the training data results in a new model that does not follow the original training points as closely, creating a smoother, more accurate function.&lt;/p&gt;

&lt;p&gt;Using an adversarial &lt;a href=&#34;https://en.wikipedia.org/wiki/Regularization_(mathematics)&#34;&gt;regularizer&lt;/a&gt; is also a technique to limit overfitting and vulnerability to adversarial examples. &lt;a href=&#34;https://arxiv.org/abs/1412.6572&#34;&gt;Goodfellow et al.&lt;/a&gt; describes a technique using the fast gradient sign method mentioned above as a regularizer. Their method generates adversarial examples using the fast gradient sign method and then trains the model with the adversarial examples. By continually updating the adversarial examples to the model, the adversarial regions are minimized. In the paper, they were able to reduce the error rate on adversarial examples from 0.94% to 0.84% using this approach. In order to reduce this error further, they also increased the model size and introduced early stopping on adversarial validation set error. Using these additional techniques they were able to drop the error rate on adversarial examples based on the FGSM from 89.4% to 17.9%. A combination of techniques introduced in training can significantly decrease the vunerability to simple adversarial attacks.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&#34;transferability-of-adversarial-samples&#34;&gt;Transferability of adversarial samples&lt;/h2&gt;

&lt;p&gt;As shown above, an adversarial sample can fool a machine-learned model to misclassify it with high confidence. In particular, the adversarial samples made to fool image classifiers show how an adversarial sample can be indistinguishable to humans from legitimate samples. Additionally, examples that evade one classifier tend to evade others. This poses a vulnerability for machine learned models, one that can be exploited with cross-evasion.&lt;/p&gt;

&lt;h4 id=&#34;learning-classifier-substitutes&#34;&gt;Learning classifier substitutes&lt;/h4&gt;

&lt;p&gt;For example, &lt;a href=&#34;https://arxiv.org/abs/1605.07277&#34;&gt;Papernot et al.&lt;/a&gt;, took advantage of the transferability of adversarial examples on a remote DNN by training a new model with their own synthetic data. By using a relatively small number of queries to label their data, they trained a substitute model which they then used to craft adversarial samples. Demonstrating transferability, 84.24% of the adversarial samples trained from the substitute model fooled the remote DNN.&lt;/p&gt;

&lt;p&gt;To explicitly demonstrate the phenomenon of both intra and cross-technique transferability, &lt;a href=&#34;https://arxiv.org/abs/1605.07277&#34;&gt;Papernot et al.&lt;/a&gt; used five different machine learning algorithms on five disjoint training sets of MNIST dataset. With this setup, adversarial samples trained from one technique on one subset of the training data fools &amp;ndash; to varying degrees &amp;ndash; models trained by a different subset of the training data as well as a different technique. Furthermore, an ensemble classifier could be fooled at a rate of up to 44%.&lt;/p&gt;

&lt;p&gt;&amp;mdash; Team Panda:&lt;br /&gt;
Christopher Geier,
Faysal Hossain Shezan,
Helen Simecek,
Lawrence Hook,
Nishant Jha&lt;/p&gt;

&lt;h4 id=&#34;references&#34;&gt;References&lt;/h4&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1412.6572.pdf&#34;&gt;[1]&lt;/a&gt; I.J. Goodfellow, J. Shlens, C. Szegedy, &amp;ldquo;Explaining
and Harnessing Adversarial Examples.&amp;rdquo; &lt;em&gt;ArXiv e-prints&lt;/em&gt;, December 2014.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1712.03141.pdf&#34;&gt;[2]&lt;/a&gt; B. Biggio, F. Roli, &amp;ldquo;Wild patterns: Ten years after the rise of
adversarial machine learning.&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1712.03141&lt;/em&gt;, 2017.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://homes.cs.washington.edu/~pedrod/papers/kdd04.pdf&#34;&gt;[3]&lt;/a&gt; N. Dalvi, P. Domingos, Mausam, S. Sanghai, D. Verma, &amp;ldquo;Adversarial classification,&amp;rdquo;&amp;rdquo; &lt;em&gt;Int’l Conf. Knowl. Disc. and Data Mining&lt;/em&gt;, 2004, pp. 99–108.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://ix.cs.uoregon.edu/~lowd/kdd05lowd.pdf&#34;&gt;[4]&lt;/a&gt; D. Lowd, C. Meek, &amp;ldquo;Adversarial learning,&amp;rdquo; &lt;em&gt;Int’l Conf. Knowl. Disc. and Data Mining&lt;/em&gt;, ACM Press, Chicago, IL, USA, 2005, pp. 641–647.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1611.03814.pdf&#34;&gt;[5]&lt;/a&gt; N. Papernot, P. McDaniel, A. Sinha, M. Wellman, &amp;ldquo;Towards the science of security and privacy in machine learning.&amp;rdquo; &lt;em&gt;IEEE European Symposium on Security and Privacy&lt;/em&gt;,
2018.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/abs/1605.07277&#34;&gt;[6]&lt;/a&gt; N. Papernot, P. Mcdaniel, I.J. Goodfellow, &amp;ldquo;Transferability in machine learning: from phenomena to black-box attacks using adversarial samples&amp;rdquo; &lt;em&gt;arXiv preprint arXiv:1605.07277&lt;/em&gt;, 2016.&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://arxiv.org/pdf/1312.6199.pdf&#34;&gt;[7]&lt;/a&gt; C. Szegedy, W. Zaremba, I. Sutskever, J Bruna, D. Erhan, I.J. Goodfellow, R. Fergus. &amp;ldquo;Intriguing properties of neural networks.&amp;rdquo; &lt;em&gt;ICLR,&lt;/em&gt; abs/1312.6199, 2014.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>First Week</title>
      <link>https://secml.github.io/first-week/</link>
      <pubDate>Sat, 20 Jan 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/first-week/</guid>
      <description>&lt;p&gt;&lt;em&gt;This message was also sent out by email.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Since not everyone has joined the slack yet, I&amp;rsquo;m sending this out by
email, but please make sure to join
&lt;a href=&#34;https://secprivml.slack.com&#34;&gt;https://secprivml.slack.com&lt;/a&gt; soon. I
will use that for future communications.&lt;/p&gt;

&lt;p&gt;I have grouped the 18 full participants in the class into three teams of six:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Team Bus:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Anant Kharkar&lt;br /&gt;
Ashley Gao&lt;br /&gt;
Atallah Hezbor&lt;br /&gt;
Joshua Holtzman&lt;br /&gt;
Mainuddin Ahmad Jonas&lt;br /&gt;
Weilin Xu&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Team Gibbon:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Aditi Narvekar&lt;br /&gt;
Austin Chen&lt;br /&gt;
Ethan Lowman&lt;br /&gt;
Guy &amp;ldquo;Jack&amp;rdquo; Verrier&lt;br /&gt;
Jialei Fu&lt;br /&gt;
Jin Ding&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Team Panda:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Bargav Jayaraman&lt;br /&gt;
Christopher Geier&lt;br /&gt;
Faysal Hossain Shezan&lt;br /&gt;
Nathaniel Grevatt&lt;br /&gt;
Nishant Jha&lt;br /&gt;
Tanmoy Sen&lt;/p&gt;

&lt;p&gt;The teams will probably adjust a bit over the next week as some people
who registered for the class didn&amp;rsquo;t submit a survey, and others may be
joining the class late, etc., but otherwise we will plan to keep with
these teams through spring break and possible re-arrange things after
that.  You can, of course, rename your team and develop your own
adversarial (but tasteful) team icon.&lt;/p&gt;

&lt;p&gt;Each week, one team will be responsible for leading the class, one
team for blogging (writing a summary of the class), and one team for
food. Everyone in the class is expected to do the preparation for each
meeting, which will usually involve reading a few papers (but may
involve other activities, at the design of the leading team).  See
&lt;a href=&#34;https://secml.github.io/teams/&#34;&gt;https://secml.github.io/teams/&lt;/a&gt; for
the description of team responsibilities.  For the first week, Team
Bus will be the leading team, Team Gibbon will be the feeding team,
and Team Panda will be the blogging team.  I have created a slack
channel for each team, and you should have an invitation to join
it. The immediate tasks for each team are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Team Bus: decide on a leader who will be responsible for coordinating your team for this week, come up with some ideas what you would like to do for Friday&amp;rsquo;s meeting, make sure there are some people who can come to my office hours on Monday (2:30pm) - if not enough people can come then, arrange an alternative time with me, and plan an exciting and worthwhile seminar for Friday.&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Team Gibbon: you are the feeding team for this week, so should select 2-3 people to take care of this. One of you should pick up a credit card from me on Thursday to pay for the food. You will be leading the seminar on 2 Feb, so should determine a leader for this, and once Team Bus posts the topic and paper for this week (which should happen on Monday), should start planning your topic. We should meet briefly after Friday&amp;rsquo;s seminar, to discuss your plans, and then a longer meeting the following Monday.&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Team Panda: you are the blogging team for this week, so should get familiar with &lt;a href=&#34;https://secml.github.io/blogging/&#34;&gt;https://secml.github.io/blogging/&lt;/a&gt;. You will be leading the seminar or 9 Feb, so should start thinking of topics you would like to lead. You should identify leaders for blogging, feeding, and leading to be ready for the next 3 weeks.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;&amp;mdash; Dave&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Starting Seminar</title>
      <link>https://secml.github.io/starting-seminar/</link>
      <pubDate>Mon, 15 Jan 2018 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/starting-seminar/</guid>
      <description>&lt;p&gt;Our first seminar meeting will be Friday, January 26 (not this Friday, which would normally be the first class day, since I will be getting back from California too late to meet this week). Meetings will be Fridays in Rice 032, 9:30am-noon.&lt;/p&gt;

&lt;p&gt;Since we&amp;rsquo;re missing the normal first meeting, I want to do as much of the organizational stuff this week to be able to have a substantive first meeting next week. This means we will group students into teams, have a team assigned to lead the first class, and announced topic and readings by next Tuesday (Jan 23).&lt;/p&gt;

&lt;p&gt;To enable this, everyone interested in participating in the class in any way should:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Submit this form by Wednesday (Jan 24): &lt;a href=&#34;https://goo.gl/forms/FqSmUNHYBPbaKsg72&#34;&gt;https://goo.gl/forms/FqSmUNHYBPbaKsg72&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Read the &lt;a href=&#34;https://secml.github.io/syllabus/&#34;&gt;course syllabus&lt;/a&gt;, and follow the directions there to join the seminar slack group.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I will work out the teams based on the form submissions, and set up the schedule for leading classes/blogging. I will have office hours on Monday (Jan 22) at 2:30pm. People from the first presenting team will be expect to meet with me then (not necessarily everyone but at least some of the team), and anyone with questions about the seminar is also welcome to come by. (If this time works for the class, we&amp;rsquo;ll keep this as my regular office hours time for the semester.)&lt;/p&gt;

&lt;p&gt;Sorry for missing the first class, but I think we&amp;rsquo;ll be able to manage most of what we would have done the first day electronically, and be able to have an effective first meeting on Jan 26.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Syllabus Posted</title>
      <link>https://secml.github.io/syllabusposted/</link>
      <pubDate>Sun, 31 Dec 2017 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/syllabusposted/</guid>
      <description>&lt;p&gt;The &lt;a href=&#34;https://secml.github.io/syllabus&#34;&gt;Syllabus&lt;/a&gt; is now posted.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title>Welcome</title>
      <link>https://secml.github.io/welcome/</link>
      <pubDate>Sun, 31 Dec 2017 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/welcome/</guid>
      <description>&lt;p&gt;This graduate-level special topics course will be offered in Spring
2018. Meetings will be &lt;strong&gt;Fridays, 9:30-noon&lt;/strong&gt; in Rice Hall 032. More
information will be posted here soon, but the seminar format will be
roughly similar to what we used to &lt;a
href=&#34;https://tlseminar.github.io&#34;&gt;TLSeminar&lt;/a&gt; last Spring.&lt;/p&gt;

&lt;p&gt;This seminar will focus on understanding the risks adversaries pose to
machine learning systems, and how to design more robust machine
learning systems to mitigate those risks.&lt;/p&gt;

&lt;p&gt;The seminar is open to ambitious undergraduate students (with
instructor permission), and to graduate students interested in
research in adversarial machine learning, privacy-preserving machine
learning, fairness and transparency in machine learning, and other
related topics.  Previous background in machine learning and security
is beneficial, but not required so long as you are willing and able to
learn some foundational materials on your own.  &lt;/p&gt; &lt;p&gt; For more
information, contact &lt;A href=&#34;https://www.cs.virginia.edu/evans&#34;&gt;David
Evans&lt;/a&gt;.&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title></title>
      <link>https://secml.github.io/resources/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/resources/</guid>
      <description>

&lt;h1 id=&#34;resources-for-adversarial-machine-learning-research&#34;&gt;Resources for Adversarial Machine Learning Research&lt;/h1&gt;

&lt;h2 id=&#34;adversarial-machine-learning-toolkits&#34;&gt;Adversarial Machine Learning Toolkits&lt;/h2&gt;

&lt;p&gt;&lt;a href=&#34;evademl.org/zoo&#34;&gt;EvadeML-Zoo&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&#34;https://github.com/tensorflow/cleverhans&#34;&gt;cleverhans&lt;/a&gt;&lt;/p&gt;

&lt;h2 id=&#34;machine-learning-systems&#34;&gt;Machine Learning Systems&lt;/h2&gt;

&lt;p&gt;&lt;a href=&#34;https://github.com/facebookresearch/Detectron&#34;&gt;Detectron&lt;/a&gt; - Facebook&amp;rsquo;s research platform for object detection research (including RetinaNet)&lt;/p&gt;
</description>
    </item>
    
    <item>
      <title></title>
      <link>https://secml.github.io/schedule/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      
      <guid>https://secml.github.io/schedule/</guid>
      <description>&lt;p&gt;See &lt;a href=&#34;https://secml.github.io/teams&#34;&gt;Teams&lt;/a&gt; for the class teams and responsibilities.&lt;/p&gt;

&lt;table&gt;
&lt;tr bgcolor=&#34;#CCC&#34;&gt;&lt;td style=&#34;text-align:center&#34; width=&#34;20%&#34;&gt;&lt;b&gt;Date&lt;/b&gt;&lt;/td&gt;&lt;td width=&#34;30%&#34; style=&#34;text-align:center&#34;&gt;&lt;b&gt;Topic&lt;/b&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; width=&#34;10%&#34;&gt;&lt;b&gt;Bus&lt;/b&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; width=&#34;10%&#34;&gt;&lt;b&gt;Gibbon&lt;/b&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; width=10%&gt;&lt;b&gt;Panda&lt;/b&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; width=12%&gt;&lt;b&gt;&lt;font size=&#34;-1&#34;&gt;Nematode&lt;/font&gt;&lt;/b&gt;&lt;/tr&gt;

&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class1&#34;&gt;Class 1&lt;/a&gt; (26 Jan)&lt;/td&gt;&lt;td&gt;Intro to Adversarial ML&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class2&#34;&gt;Class 2&lt;/a&gt; (2 Feb)&lt;/td&gt;&lt;td&gt;Privacy and ML&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class3&#34;&gt;Class 3&lt;/a&gt; (9 Feb)&lt;/td&gt;&lt;td&gt;Adversarial Examples&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class4&#34;&gt;Class 4&lt;/a&gt; (16 Feb)&lt;/td&gt;&lt;td&gt;Privacy in Action&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class5&#34;&gt;Class 5&lt;/a&gt; (23 Feb)&lt;/td&gt;&lt;td&gt;Audio, Text, Faces&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class6&#34;&gt;Class 6&lt;/a&gt; (2 Mar)&lt;/td&gt;&lt;td&gt;Robustness&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td colspan=6 bgcolor=&#34;#66EEAA&#34; style=&#34;text-align:center&#34;&gt;Spring Break&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class7&#34;&gt;Class 7&lt;/a&gt; (16 Mar)&lt;/td&gt;&lt;td&gt;Bias and Discrimination&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class8&#34;&gt;Class 8&lt;/a&gt; (23 Mar)&lt;/td&gt;&lt;td&gt;Testing&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;/tr&gt;

&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class9&#34;&gt;Class 9&lt;/a&gt; (30 Mar)&lt;/td&gt;&lt;td&gt;Malware&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class10&#34;&gt;Class 10&lt;/a&gt; (6 Apr)&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class11&#34;&gt;Class 11&lt;/a&gt; (13 Apr)&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;/tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class12&#34;&gt;Class 12&lt;/a&gt; (20 Apr)&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34; &gt;Food&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCC&#34;&gt;&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#DEF&#34;&gt;Blog&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  bgcolor=&#34;#CCDD55&#34;&gt;Lead&lt;/td&gt;&lt;/tr&gt;

&lt;tr&gt;&lt;td&gt;&lt;a href=&#34;https://secml.github.io/class13&#34;&gt;Class 13&lt;/a&gt; (26 Apr)&lt;/td&gt;&lt;td style=&#34;text-align:center&#34;  colspan=5 bgcolor=&#34;#FF3&#34; style=&#34;text-align:center&#34;&gt;Mini-Conference&lt;/td&gt;&lt;/tr&gt;

&lt;/table&gt;
</description>
    </item>
    
  </channel>
</rss>