I think a majority of people do care, but there's not really a way for them to express that in the market. You can't just buy the more expensive product and trust that it's going to be more ethically produced.
It's a societal problem that can't be solved by individual consumers.
Non-regulatory solutions also doesn't empower the market to optimize cost in a productive way (which would cut into profits). In the current economy for the vast majority of products we have two options 1) a cheap option made in horrendous inhumane conditions and 2) a luxury good made in a process that's inherently limited in aim and scope. Globalism basically exports the suffering of the most unregulated markets to the entire planet.