I would say that one of the goals of note taking is to make it so that you don't need to go back to them. If you write notes about a book you read, not copy-pasting the exact quotes, but writing your understanding, you'll be more likely to remember it later. And if, at some point you want to remember more, then you actually go to the note and check.