Heatmapping.org

Upcoming Events Past Events

2019-10-27 Talk at ICCV 2019 Workshop on Interpretating and Explaining Visual AI Models in Seoul, Korea

2019-07-23 Tutorial on "Interpretable & Transparent Deep Learning" at EMBC 2019 in Berlin, Germany

2019-06-16 Talk at CVPR 2019 Workshop on Explainable AI in Long Beach, CA, USA

2018-10-12 Keynote at the 2018 International Explainable AI Symposium in Seoul, Korea

2018-10-07 Tutorial on "Interpretable Deep Learning: Towards Understanding & Explaining Deep Neural Networks" at ICIP 2018 in Athens, Greece

2018-09-16 Tutorial on "Interpretable Machine Learning" at MICCAI 2018 in Granada, Spain

2018-06-18 Tutorial on "Interpreting and Explaining Deep Models in Computer Vision" at CVPR 2018 in Salt Lake City, USA

2017-12-09 Workshop on "Interpreting, Explaining and Visualizing Deep Learning" at NIPS 2017 in Long Beach, CA

2017-12-04 Tutorial on "Understanding Deep Neural Networks and their Predictions" at WIFS 2017 in Rennes, France

2017-11-30 Talk at the CoSIP Intense Course on Deep Learning in Berlin, Germany

2017-09-12 Tutorial on "Interpretable Machine Learning" at GCPR 2017 in Basel, Switzerland

2017-08-28 Class on "Interpretable ML" at DTU Summer School on Advanced Topics in Machine Learning in Copenhagen, Denmark

2017-03-20 Demonstration at CeBIT 2017 in Hannover, Germany

2017-03-05 Tutorial at ICASSP 2017 on "Methods for Interpreting and Understanding Deep Neural Networks" in New Orleans, USA

2016-12-09 Presentation at the NIPS 2016 Workshop on Interpretable ML for Complex Systems in Barcelona, Spain

2016-11-24 ACCV 2016 Workshop on Interpretation and Visualization of Deep Neural Nets in Taipei, Taiwan

2016-09-26 The LRP Toolbox presented at the ICIP Visual Technology Showcase in Phoenix, Arizona

2016-09-06 ICANN 2016 Workshop on Machine Learning and Interpretability in Barcelona, Spain

This webpage aims to regroup publications and software produced as part of a joint project at Fraunhofer HHI, TU Berlin and SUTD Singapore on developing new method to understand nonlinear predictions of state-of-the-art machine learning models.

Machine learning models, in particular deep neural networks (DNNs), are characterized by very high predictive power, but in many case, are not easily interpretable by a human. Interpreting a nonlinear classifier is important to gain trust into the prediction, and to identify potential data selection biases or artefacts.

The project studies in particular techniques to decompose the prediction in terms of contributions of individual input variables such that the produced decomposition (i.e. explanation) can be visualized in the same way as the input data.

Interactive LRP Demos

Draw a handwritten digit and see the heatmap being formed in real-time. Create your own heatmap for natural images or text. These demos are based on the Layer-wise Relevance Propagation (LRP) technique by Bach et al. (2015).

MNIST: A simple LRP demo based on a neural network that predicts handwritten digits and was trained using the MNIST data set.

Caffe: A more complex LRP demo based on a neural network implemented using Caffe. The neural network predicts the contents of the picture.

Text: A LRP demo that explains classification on natural language. The neural network predicts the type of document.

How and Why LRP ?

Layer-wise Relevance Propagation (LRP) is a method that identifies important pixels by running a backward pass in the neural network. The backward pass is a conservative relevance redistribution procedure, where neurons that contribute the most to the higher-layer receive most relevance from it. The LRP procedure is shown graphically in the figure below.

The method can be easily implemented in most programming languages and integrated to existing neural network frameworks. The propagation rules used by LRP can for many architectures, including deep rectifier networks or LSTMs, be understood as a Deep Taylor Decomposition of the prediction.

LRP Software

Tutorials

CVPR Tutorial 2018

Video Part 1
Video Part 2
Introduction
Methods
Applications 1
Applications 2
Tutorial on Implementing LRP
ICCV 2019 XAI Workshop Keynote (Website, Slides)
Explainable ML, Applications
EMBC 2019 Tutorial (Slides: 1-Intro, 2-Methods, 3-Evaluation, 4-Applications)
Explainable ML, Medical Applications
Northern Lights Deep Learning Workshop Keynote (Website | Slides)
Explainable ML, Applications
2018 Int. Explainable AI Symposium Keynote (Website | Slides)
Explainable ML, Applications
ICIP 2018 Tutorial (Website | Slides: 1-Intro, 2-Methods, 3-Evaluation, 4-Applications)
Explainable ML, Applications
MICCAI 2018 Tutorial (Website | Slides)
Explainable ML, Medical Applications
Talk at Int. Workshop ML & AI 2018 (Slides)
Deep Taylor Decomposition, Validating Explanations
WCCI 2018 Keynote (Slides)
Explainable ML, LRP, Applications
GCPR 2017 Tutorial (Slides)
ICASSP 2017 Tutorial (Slides 1-Intro, 2-Methods, 3-Applications)

Publications

Edited Books

W Samek, G Montavon, A Vedaldi, LK Hansen, KR Müller (Eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
Springer LNCS 11700, 2019

Highlight Papers

S Lapuschkin, S Wäldchen, A Binder, G Montavon, W Samek, KR Müller. Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
Nature Communications, 10:1096, 2019 [preprint, bibtex]

Tutorial / Overview Papers

G Montavon, W Samek, KR Müller. Methods for Interpreting and Understanding Deep Neural Networks
Digital Signal Processing, 73:1-15, 2018 [preprint, bibtex]
W Samek, T Wiegand, KR Müller. Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
ITU Journal: ICT Discoveries - Special Issue 1 - The Impact of AI on Communication Networks and Services, 1(1):39-48, 2018 [preprint, bibtex]
W Samek, KR Müller. Towards Explainable Artificial Intelligence
in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS 11700, 2019 [preprint, bibtex]
G Montavon, A Binder, S Lapuschkin, W Samek, KR Müller. Layer-Wise Relevance Propagation: An Overview
in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS 11700, 2019 [preprint, bibtex]

Methods Papers

S Bach, A Binder, G Montavon, F Klauschen, KR Müller, W Samek. On Pixel-wise Explanations for Non-Linear Classifier Decisions by Layer-wise Relevance Propagation
PLOS ONE, 10(7):e0130140, 2015 [preprint, bibtex]
G Montavon, S Lapuschkin, A Binder, W Samek, KR Müller. Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
Pattern Recognition, 65:211–222, 2017 [preprint, bibtex]
L Arras, J Arjona, M Widrich, G Montavon, M Gillhofer, KR Müller, S Hochreiter, W Samek. Explaining and Interpreting LSTMs
in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS 11700, 2019 [preprint, bibtex]
L Arras, G Montavon, KR Müller, W Samek. Explaining Recurrent Neural Network Predictions in Sentiment Analysis
EMNLP Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, 159-168, 2017 [preprint, bibtex]
M Kohlbrenner, A Bauer, S Nakajima, A Binder, W Samek, S Lapuschkin. Towards best practice in explaining neural network decisions with LRP
arXiv:1910.09840, 2019
A Binder, G Montavon, S Lapuschkin, KR Müller, W Samek. Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers
Artificial Neural Networks and Machine Learning – ICANN 2016, Part II, Lecture Notes in Computer Science, Springer-Verlag, 9887:63-71, 2016 [preprint, bibtex]
PJ Kindermans, KT Schütt, M Alber, KR Müller, D Erhan, B Kim, S Dähne. Learning how to explain neural networks: PatternNet and PatternAttribution
International Conference on Learning Representations (ICLR), 2018
L Rieger, P Chormai, G Montavon, LK Hansen, KR Müller. Structuring Neural Networks for More Explainable Predictions
in Explainable and Interpretable Models in Computer Vision and Machine Learning, 115-131, Springer SSCML, 2018

Explaining Unsupervised Learning

J Kauffmann, M Esders, G Montavon, W Samek, KR Müller. From Clustering to Cluster Explanations via Neural Networks
arXiv:1906.07633, 2019
J Kauffmann, KR Müller, G Montavon. Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
Pattern Recognition, 107198, 2020 [preprint]

Evaluation of Explanations

W Samek, A Binder, G Montavon, S Bach, KR Müller. Evaluating the Visualization of What a Deep Neural Network has Learned
IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-2673, 2017 [preprint, bibtex]
L Arras, A Osman, KR Müller, W Samek. Evaluating Recurrent Neural Network Explanations
ACL Workshop on BlackboxNLP, 2019 [preprint, bibtex]
G Montavon. Gradient-Based Vs. Propagation-Based Explanations: An Axiomatic Comparison
in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS 11700, 2019 [bibtex]

Software Papers

M Alber, S Lapuschkin, P Seegerer, M Hägele, KT Schütt, G Montavon, W Samek, KR Müller, S Dähne, PJ Kindermans iNNvestigate neural networks!
Journal of Machine Learning Research, 20(93):1−8, 2019 [preprint, bibtex]
M Alber. Software and Application Patterns for Explanation Methods
in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS 11700, 2019 [bibtex]
S Lapuschkin, A Binder, G Montavon, KR Müller, W Samek The Layer-wise Relevance Propagation Toolbox for Artificial Neural Networks
Journal of Machine Learning Research, 17(114):1−5, 2016 [preprint, bibtex]

Application to Sciences

I Sturm, S Bach, W Samek, KR Müller. Interpretable Deep Neural Networks for Single-Trial EEG Classification
Journal of Neuroscience Methods, 274:141–145, 2016 [preprint, bibtex]
A Binder, M Bockmayr, M Hägele, S Wienert, D Heim, K Hellweg, A Stenzinger, L Parlow, J Budczies, B Goeppert, D Treue, M Kotani, M Ishii, M Dietel, A Hocke, C Denkert, KR Müller, F Klauschen. Towards computational fluorescence microscopy: Machine learning-based integrated prediction of morphological and molecular tumor profiles
arXiv:1805.11178, 2018
F Horst, S Lapuschkin, W Samek, KR Müller, WI Schöllhorn. Explaining the Unique Nature of Individual Gait Patterns with Deep Learning
Scientific Reports, 9:2391, 2019 [preprint, bibtex]
AW Thomas, HR Heekeren, KR Müller, W Samek. Analyzing Neuroimaging Data Through Recurrent Deep Learning Models
Frontiers in Neuroscience, 13:1321, 2019 [preprint, bibtex]

Application to Text

L Arras, F Horn, G Montavon, KR Müller, W Samek. "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
PLOS ONE, 12(8):e0181142, 2017 [preprint, bibtex]
L Arras, G Montavon, KR Müller, W Samek. Explaining Recurrent Neural Network Predictions in Sentiment Analysis
EMNLP Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, 159-168, 2017 [preprint, bibtex]
L Arras, F Horn, G Montavon, KR Müller, W Samek. Explaining Predictions of Non-Linear Classifiers in NLP
ACL Workshop on Representation Learning for NLP, 1-7, 2016 [preprint, bibtex]
F Horn, L Arras, G Montavon, KR Müller, W Samek. Exploring text datasets by visualizing relevant words
arXiv:1707.05261, 2017

Application to Images & Faces

S Lapuschkin, A Binder, G Montavon, KR Müller, W Samek. Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2912-2920, 2016 [preprint, bibtex]
S Lapuschkin, A Binder, KR Müller, W Samek. Understanding and Comparing Deep Neural Networks for Age and Gender Classification
IEEE International Conference on Computer Vision Workshops (ICCVW), 1629-1638, 2017 [preprint, bibtex]
C Seibold, W Samek, A Hilsmann, P Eisert. Accurate and Robust Neural Networks for Security Related Applications Exampled by Face Morphing Attacks
arXiv:1806.04265, 2018
S Bach, A Binder, KR Müller, W Samek. Controlling Explanatory Heatmap Resolution and Semantics via Decomposition Depth
Proceedings of the IEEE International Conference on Image Processing (ICIP), 2271-2275, 2016 [preprint, bibtex]
A Binder, S Bach, G Montavon, KR Müller, W Samek. Layer-wise Relevance Propagation for Deep Neural Network Architectures
Proceedings of the 7th International Conference on Information Science and Applications (ICISA), 6679:913-922, Springer Singapore, 2016 [preprint, bibtex]
F Arbabzadah, G Montavon, KR Müller, W Samek. Identifying Individual Facial Expressions by Deconstructing a Neural Network
Pattern Recognition - 38th German Conference, GCPR 2016, Lecture Notes in Computer Science, 9796:344-354, 2016 [preprint, bibtex]

Application to Video

C Anders, G Montavon, W Samek, KR Müller. Understanding Patch-Based Learning of Video Data by Explaining Predictions
in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer LNCS 11700, 2019 [preprint, bibtex]
V Srinivasan, S Lapuschkin, C Hellge, KR Müller, W Samek. Interpretable human action recognition in compressed domain
Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1692-1696, 2017 [preprint, bibtex]

Application to Speech

S Becker, M Ackermann, S Lapuschkin, KR Müller, W Samek. Interpreting and Explaining Deep Neural Networks for Classification of Audio Signals
arXiv:1807.03418, 2018

Application to Neural Network Pruning

S Yeom, P Seegerer, S Lapuschkin, S Wiedemann, KR Müller, W Samek. Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
arXiv:1912.08881, 2019

Application to Dataset Denoising

CJ Anders, T Marinc, D Neumann, W Samek, KR Müller, S Lapuschkin. Analyzing ImageNet with Spectral Relevance Analysis: Towards ImageNet un-Hans'ed
arXiv:1912.11425, 2019

Downloads

BVLC Model Zoo Contributions

Pascal VOC 2012 Multilabel Model (see paper): [caffemodel] [prototxt]
Age and Gender Classification Models (see paper): [data and models]