Exciting research. Not my area of expertise. An opinion:
> without any knowledge of the game rules
I'd prefer 1000 times an AI that can explain to me why an opposite-colored bishop ending is drawish or why knights are more valuable near the center, and can come up with those and more new concepts/relationships/models on its own (regardless of whether we have given it the game rules or not), than a black box that is excellent at beating you at chess but you can't understand or trust. Adversarial examples (false positives) are supporters for this preference.
There are ways to reverse engineer neural nets to try and understand the reasoning but AFAIK none of them are very good.
For example, I went to a SAS/STAT course a few years ago and one of the exercises was to train a simple neural network and then use it as input to generate a decision tree, that would (with pretty good accuracy, around 95% or so) explain the choices made by the neural network. I think the given scenario was that the neural network was used to decide who to send marketing emails to, and the head of marketing wanted to know why it was choosing certain customers.
The problem with this was that it didn't offer insight into why decisions were made, it could only show what variable was being branched on at a given point in the tree. Also while it worked with this simple example, more complex NNs generated huge decision trees that were not usefull in practice.
Generally, it's a superhuman task. No one yet came up with an explanation of how to tell apart a picture of a kitten and a picture of a puppy that doesn't rely on our common low-level visual processing.
That is the explanation mechanism should take into account what kind of explanations humans consider understandable. For tasks that can be represented as mathematical structures (like two bishop ending), it's probably simple enough. For tasks that we don't really know how we do them (vision, hearing, locomotion planning), the explanation mechanism will have to learn somehow what we consider a good explanation and somehow translate internal workings of decision network and its own network (we'll want to know why it explains it like it does, right?) into them.
Since you left 'valuable' intentionally vague, I asked myself what the value of a piece of knowledge would be.
So I quickly came up with a simple model. Just take the integral of the value you get from this piece of knowledge over time, I thought. This helps you measure if a one-time payoff of knowledge A is better than the repeated payoff of knowledge B. Then I realized gauging value is more nuanced. How to model the value of things that are less tangible like insurance? The Heimlich maneuver may have a really high value in the case that my friend chokes, but very low value otherwise.
But this kind of knowledge is harder to focus on when we measure our knowledge by some simple value model. Surely you didn't expect an answer to "the most valuable thing you can learn in an hour" to be about what the best kind of insurance you should get?
> without any knowledge of the game rules
I'd prefer 1000 times an AI that can explain to me why an opposite-colored bishop ending is drawish or why knights are more valuable near the center, and can come up with those and more new concepts/relationships/models on its own (regardless of whether we have given it the game rules or not), than a black box that is excellent at beating you at chess but you can't understand or trust. Adversarial examples (false positives) are supporters for this preference.