This is true. Indeed, the true question usually revolves around transfer. That is, as your brain changes in response to a certain memory task (e.g. recognition of visual objects) to get better at it, will those changes transfer benefits to other types of memory tasks (e.g. recall of visual objects), or tasks which have a memory component (e.g. navigation).
Transfer has been the bane of most cognitive training paradigms. Improvements are seen in the same task, but do not seem to transfer to conceptually related tasks. One idea to overcome problems in transfer has been to identify tasks which inordinately rely on core cognitive processes of memory (e.g. recollection vs. familiarity; Yonelinas, 2002), or relational binding (Konkel & Cohen, 2009). Another idea might be to train using memory strategies, like loci. Typically that research looks to whether using a particular strategy benefits memory, and frequently, does that strategy work in a particular population with poorer memory (e.g. children, adults with Alzheimer)
I diverted my attention to lower hanging fruit in my Ph.D., so stopped following that literature closely. Overall I think that unless you already exercise regularly, exercising will have a larger effect on cognitive ability than any task-based cognitive training regimen.
Transfer has been the bane of most cognitive training paradigms. Improvements are seen in the same task, but do not seem to transfer to conceptually related tasks. One idea to overcome problems in transfer has been to identify tasks which inordinately rely on core cognitive processes of memory (e.g. recollection vs. familiarity; Yonelinas, 2002), or relational binding (Konkel & Cohen, 2009). Another idea might be to train using memory strategies, like loci. Typically that research looks to whether using a particular strategy benefits memory, and frequently, does that strategy work in a particular population with poorer memory (e.g. children, adults with Alzheimer)