You can perform extremely good compression this way, but the computational and energy cost would be prohibitive.

There are even more traditional tricks that don't make it in things like H.265 because it is too costly.

Here is my work, where I do use semantic information to achieve compression (rather improve JPEG). This is not an end to end compression like Google's work, but just incorporating semantic knowledge into compression. I am still trying to clean up the code before I make arxiv/github submission, but since you are interested here is the link http://gpgpu.cs-i.brandeis.edu/semantic_jpeg.pdf

I understood that to be the tech behind "Silicon Valley".