The maximum energy you can extract from a unit of heat is based on the temperature ratio of your hot object and your cold object.

For light coming from the sun, the hot object is 6000K and the cold object is around 300K. So about 95% of the energy is available to attempt to capture. 94% if your solar panel gets very hot.

The infrared coming off local hot objects is almost useless in comparison. If you can keep your infrared-catcher perfectly cooled with an underground loop, it will be at 290K. If you have an object that's all the way up at 150F, that's still only 340K and less than 15% of that energy could possibly be captured.