You can, and people do that today. It has limitations though:
* The data must be sorted in order for Git to provide good diffs
* It does not scale very well. On my machine, Git refuses to diff files over 1GB (maybe there is a setting for that)
* You must clone the entire repository onto your machine to work with it
* There is no programmatic API -- you must work with the data and changes as text and line diffs
However, 1 question I have is:
Couldn't you just put a CSV/JSON file(s) behind VCS?
Eg. Drop my CSV/JSON file(s) onto github.com and then it will be version-controlled ?