Feature/security mia ml leaks

docs/source/gallery/tutorials/security.rst

@@ -22,3 +22,7 @@ Privacy Risk Score
 Mitigation
 ----------
 - `Anonymization <security/mitigating_privacy_tutorials/anonymization_algorithm.ipynb>`_
+
+Attackers
+---------
+- `Membership Inference ML-leaks <security/attackers/ml_leaks.ipynb>`_

docs/source/gallery/tutorials/security/attackers/ml_leaks.ipynb

@@ -0,0 +1,239 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8f36a4eb",
+   "metadata": {},
+   "source": [
+    "# Membership inference attack of a binary classification Model\n",
+    "\n",
+    "In this notebook, we will evaluate the security of a binary classification model trained on the Adult dataset. We will use the ML-leaks method to perform membership inference attack (MIA) to train an attacker model and measure the model's security against this attacker using traditional metrics."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "e7d2e610",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from holisticai.security.attackers import MLleaks\n",
+    "from holisticai.datasets import load_dataset\n",
+    "from sklearn.ensemble import RandomForestClassifier\n",
+    "from sklearn.metrics import accuracy_score\n",
+    "from sklearn import metrics"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a7148d07",
+   "metadata": {},
+   "source": [
+    "## Loading the dataset\n",
+    "\n",
+    "We will use the Adult dataset, which contains census data. The target variable is whether a person's income exceeds $50K/year, and the protected attribute we will consider is 'sex'. For time constraints, we will only use a small subset of the data for testing the model and use the attackers.\n",
+    "\n",
+    "Following the MIA pipeline of this method, we will assume that the attacker has a dataset that comes from the same distribution as the training dataset. This dataset will be used to train a shadow model that will be used to mimic the target model. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "5a5c0dae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset = load_dataset('adult', protected_attribute='sex', preprocessed=True)\n",
+    "train_test = dataset.train_test_split(test_size=0.5, random_state=42)\n",
+    "target = train_test['train'].train_test_split(test_size=0.5, random_state=42)\n",
+    "shadow = train_test['test'].train_test_split(test_size=0.5, random_state=42)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "b16eb921",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Training set size: 11305\n",
+      "Test set size: 11306\n",
+      "Shadow Training set size: 11305\n",
+      "Shadow Test set size: 11306\n"
+     ]
+    }
+   ],
+   "source": [
+    "X_target_train = target['train']['X']\n",
+    "y_target_train = target['train']['y']\n",
+    "X_target_test = target['test']['X']\n",
+    "y_target_test = target['test']['y']\n",
+    "\n",
+    "X_shadow_train = shadow['train']['X']\n",
+    "y_shadow_train = shadow['train']['y']\n",
+    "X_shadow_test = shadow['test']['X']\n",
+    "y_shadow_test = shadow['test']['y']\n",
+    "\n",
+    "print(\"Training set size:\", X_target_train.shape[0])\n",
+    "print(\"Test set size:\", X_target_test.shape[0])\n",
+    "\n",
+    "print(\"Shadow Training set size:\", X_shadow_train.shape[0])\n",
+    "print(\"Shadow Test set size:\", X_shadow_test.shape[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "def203ae",
+   "metadata": {},
+   "source": [
+    "## Training the target model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "194cf6d2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Target Model Performance:\n",
+      "Accuracy: 0.8439766495666018\n"
+     ]
+    }
+   ],
+   "source": [
+    "target_model = RandomForestClassifier(random_state=42)\n",
+    "target_model.fit(X_target_train, y_target_train)\n",
+    "y_target_pred = target_model.predict(X_target_test)\n",
+    "print(\"Target Model Performance:\")\n",
+    "print(\"Accuracy:\", accuracy_score(y_target_test, y_target_pred))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2a4c09d1",
+   "metadata": {},
+   "source": [
+    "## Setting up the membership inference attack\n",
+    "\n",
+    "We will use the ML-leaks method to perform membership inference attack (MIA) presented by Salem et al. in 2018. The method relaxes the assumption that the attacker uses multiple shadow models to train the attacker model. Instead, it uses a single shadow model to get the confidence scores, generate and attacker dataset and use it to train the attacker model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "1744b648",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Task: binary\n"
+     ]
+    }
+   ],
+   "source": [
+    "tgt_dataset = ((X_target_train, y_target_train), (X_target_test, y_target_test))\n",
+    "sdw_dataset = ((X_shadow_train, y_shadow_train), (X_shadow_test, y_shadow_test))\n",
+    "\n",
+    "mia_attacker = MLleaks(target_model, tgt_dataset, sdw_dataset)\n",
+    "train_attacker_data, test_attacker_data = mia_attacker.generate_attack_dataset()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "e423c0b6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "X_mia_train, y_mia_train = train_attacker_data\n",
+    "X_mia_test, y_mia_test = test_attacker_data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "9e0d0a24",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Attacker Model Performance:\n",
+      "Accuracy: 0.6052363893680067\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Train the Attacker Model\n",
+    "attacker_model = mia_attacker.fit()\n",
+    "y_mia_test_pred = attacker_model.predict(X_mia_test)\n",
+    "print(\"Attacker Model Performance:\")\n",
+    "print(\"Accuracy:\", accuracy_score(y_mia_test, y_mia_test_pred))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "bcb13897",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Attacker Precision: 0.5752705182560274\n",
+      "Attacker Recall: 0.8041574524546661\n",
+      "Attacker F1 Score: 0.6707245093699277\n"
+     ]
+    }
+   ],
+   "source": [
+    "precision = metrics.precision_score(y_mia_test, y_mia_test_pred)\n",
+    "recall = metrics.recall_score(y_mia_test, y_mia_test_pred)\n",
+    "f1 = metrics.f1_score(y_mia_test, y_mia_test_pred)\n",
+    "print(\"Attacker Precision:\", precision)\n",
+    "print(\"Attacker Recall:\", recall)\n",
+    "print(\"Attacker F1 Score:\", f1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a63fd318",
+   "metadata": {},
+   "source": [
+    "From these results, we can see that the attacker model achieves a precision of 0.575 and a recall of 0.805. The F1 score is 0.67, which indicates that the attacker model is able to identify the members of the target model with a reasonable accuracy. "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "testing",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/source/reference/index.rst

@@ -67,6 +67,8 @@ Security in machine learning involves practices and measures to protect AI model
      - Metrics for evaluating security in classification and regression tasks.
    * - `Mitigation <security/mitigation.html>`_
      - Strategies for enhancing security and robustness in learning tasks.
+   * - `Attackers <security/attackers.html>`_
+     - Techniques and strategies to simulate attacks and test model security.
 
 Robustness
 ----------

docs/source/reference/security/attackers.rst

@@ -0,0 +1,15 @@
+:py:mod:`holisticai.security.attackers`
+=========================================
+
+.. automodule:: holisticai.security.attackers
+    :no-members:
+    :no-inherited-members:
+
+**Membership inference attack**
+
+.. autosummary::
+    :nosignatures:
+    :template: function.rst
+    :toctree: .generated/
+
+    MLleaks

src/holisticai/security/attackers/__init__.py

@@ -0,0 +1,3 @@
+from holisticai.security.attackers.membership_inference.ml_leaks import MLleaks
+
+__all__ = ["MLleaks"]