If you believe in the Cellular Automaton Interpretation of Quantum Mechanics by the Nobel laureate Gerard 't Hooft (which is somewhat controversial among physicists), building a sufficiently large quantum computer will probably be impossible. His book is available for free:
According to 't Hooft himself (source: https://physicstoday.scitation.org/do/10.1063/PT.6.4.2017071...) "The response [by fellow researchers] has been very mixed. Many other researchers are clearly very skeptical. They should be, because there are important unanswered questions. Others have expressed their interest and support. What concerns me is that I haven’t yet found colleagues who completely understand my approach. And also, of course, I don’t know what they say behind my back.".
It depends on what you mean by "doubts". In the strict mathematical proof sense, we don't know if quantum computers are more powerful than classical computers at any task. In other words, like many conjectures in complexity theory (like P!=NP), the statement BQP!=P is still a conjecture. But computer scientists have good reasons to think that these conjectures will be true, after many decades of grappling with these problems and failing to prove otherwise. Therefore, in this informal expert-feeling sense of the word, there are not many "doubts".
