What is Explainable AI — Permutation Feature Importance using Tensorflow

How do we trust that AI is making good decisions? How do we affirm the decisions of our typical deep neural networks? How can AI explain itself?

Explainable AI (XAI) refers to the development of artificial intelligence (AI) that can be easily understood by humans.

The goal of XAI is to create AI systems that are transparent, interpretable, and accountable, so that humans can trust and rely on them.

In this article, we will discuss the importance of explainable AI and provide a sample TensorFlow code for building an interpretable model.

Importance of Explainable AI

As AI technologies become more prevalent in our daily lives, it is crucial to ensure that these systems are transparent and accountable. With explainable AI, users can understand how an AI model arrived at a decision or prediction, which can help to build trust and credibility in the system. Moreover, explainable AI can help to identify biases and errors in the model, and provide insights for improving its performance.

Building an Interpretable TensorFlow Model

To build an interpretable TensorFlow model, we can use a technique called feature importance. Feature importance is a method for understanding which features in the dataset have the most impact on the model’s predictions. There are several ways to calculate feature importance, including permutation feature importance and SHAP (SHapley Additive exPlanations).

Here’s an example of how to calculate feature importance using the permutation feature importance technique:

import tensorflow as tf
from tensorflow.keras.datasets import mnist

# Load the MNIST dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Reshape the input data to be a vector
x_train = x_train.reshape((x_train.shape[0], -1))
x_test = x_test.reshape((x_test.shape[0], -1))

# Define the model architecture
model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', 
    metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train, epochs=10)

# Calculate feature importance
import numpy as np

def feature_importance(model, x_test):
    baseline = model.evaluate(x_test, y_test)[1]
    importance = []
    for i in range(x_test.shape[1]):
        x_permuted = x_test.copy()
        np.random.shuffle(x_permuted[:, i])
        score = model.evaluate(x_permuted, y_test)[1]
        importance.append(baseline - score)
    return importance

importance = feature_importance(model, x_test)
print(importance)

In this code, we first load the MNIST dataset and reshape the input data to be a vector. We then define a simple neural network with two dense layers and compile it using the Adam optimizer and sparse categorical cross-entropy loss.

We train the model on the training data for 10 epochs.

After training the model, we define a function called feature_importance that takes in the trained model and the test data, and returns a list of feature importance scores. In the function, we iterate through each feature in the test data and randomly permute its values. We then evaluate the model on the permuted data and subtract the resulting accuracy score from the baseline accuracy (i.e., the accuracy on the original data) to calculate the feature importance score.

Finally, we call the feature_importance function on the test data and print the resulting feature importance scores.

Conclusion

Explainable AI is becoming increasingly important as AI systems become more prevalent in our daily lives. By building interpretable models, we can help to build trust and credibility in AI systems, as well as identify biases and errors in the models. In this article, we demonstrated how to calculate feature importance using the permutation feature importance.