Expert systems from the 70s and 80s had a capability similar to this. They could explain how they reached a conclusion by reporting the rules they used to get there. The problem was that interviewing experts and coming up with a huge rules database was a ton of work and didn't scale very well.

Maybe the next direction in AI will be to bridge the gap between expert systems and black box models?