{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Building confidence on explainability methods\n", "\n", "Local explainability methods are recent, they keep being improved and research is very active in the field. Thus, one needs to be careful when interpreting those methods' outputs. \n", "\n", "Depending on the tasks and data, explainability methods may give different results. The purpose of the metrics presented below is to assess the degree of confidence in these cases. We will answer the following questions: \n", "\n", "- Do different explainability methods give similar explanations on average ? (**Consistency**)\n", "\n", "- Is the explanation similar for similar instances ? (**Local stability**)\n", "\n", "- Do a few features drive the model ? (**Approximation**)\n", "\n", "This short tutorial presents an example of how those 3 metrics could be used on a classification case.\n", "\n", "We used Kaggle's [Titanic](https://www.kaggle.com/c/titanic/data) dataset" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "from category_encoders import OrdinalEncoder\n", "from sklearn.ensemble import ExtraTreesClassifier\n", "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Building Supervized Model\n", "\n", "Let's start by loading a dataset and building a model that we will try to explain right after.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load Titanic data" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from shapash.data.data_loader import data_loading\n", "titanic_df, titanic_dict = data_loading('titanic')\n", "del titanic_df['Name']\n", "y_df=titanic_df['Survived'].to_frame()\n", "X_df=titanic_df[titanic_df.columns.difference(['Survived'])]" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | Survived | \n", "Pclass | \n", "Sex | \n", "Age | \n", "SibSp | \n", "Parch | \n", "Fare | \n", "Embarked | \n", "Title | \n", "
---|---|---|---|---|---|---|---|---|---|
PassengerId | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
1 | \n", "0 | \n", "Third class | \n", "male | \n", "22.0 | \n", "1 | \n", "0 | \n", "7.25 | \n", "Southampton | \n", "Mr | \n", "
2 | \n", "1 | \n", "First class | \n", "female | \n", "38.0 | \n", "1 | \n", "0 | \n", "71.28 | \n", "Cherbourg | \n", "Mrs | \n", "
3 | \n", "1 | \n", "Third class | \n", "female | \n", "26.0 | \n", "0 | \n", "0 | \n", "7.92 | \n", "Southampton | \n", "Miss | \n", "
4 | \n", "1 | \n", "First class | \n", "female | \n", "35.0 | \n", "1 | \n", "0 | \n", "53.10 | \n", "Southampton | \n", "Mrs | \n", "
5 | \n", "0 | \n", "Third class | \n", "male | \n", "35.0 | \n", "0 | \n", "0 | \n", "8.05 | \n", "Southampton | \n", "Mr | \n", "