How many times have you look into webkit or gecko 
  source code to fix layout problems?

That's not really the point. Let's use the Adobe Flex Framework as an example. If you bought into Flex 5 years ago then you're probably starting to feel a bit nervous. The reasons being:

  1. Adobe are themselves signaling that 
  HTML is the future by releasing HTML5 tools (Edge)
  2. Adobe have abandoned mobile Flash development
  3. Developer mind-share is firmly in the HTML camp.

Now, if the full adobe/flash stack were open source and there were multiple companies working on it, it would be less of an issue as the technology you invested into would continue to be supported. The fact that you, personally are not working on the flash/flex core would be irrelevant, as is the case with HTML5 and the open web.

  If you think a layout engine is complex enough that 
  need to be open sourced, you are doing it wrong.

I'm at a loss trying to respond to this. If you think writing a layout engine from scratch is easy then you simply don't understand the problem very well. Even writing a good text layout engine is very hard to get right, and text layout is only a subset of the whole problem. Not only that, but layout is an orthogonal problem to the one you're trying to solve (unless you're writing a browser or word processor), so the time you sink into it is wasted.

  native app development (Visual Studio, XCode) layouts are much 
  simpler and more powerful

It's simpler because the complexity is abstracted away from you by the IDE/tools. The actual (hidden) application logic driving the layouts is just as complex if not more so.

  The problem with HTML and CSS is that they are either 
  document flow based or absolute position.

Absolutely agree with you here. The document paradigm has been abused horribly to allow for complex layout on the web, and its roots are painfully evident - it certainly wasn't meant for this type of work.

  It's easier to do it right in native ways. There are much richer containers,
  control groups and docking options.

Agree here too. Having said that though there are great advances being made, the flexible box model being one of them.

  Oh wow, Javascript for layout, right.

That's sarcasm not a rebuttal :)

Going back to the original point, I think that CSS for native layout may be an OK idea, CSS certainly has it share of strengths and weaknesses. One thing this approach has in its favor is a low barrier to entry for web developers which may help bridge the designer/developer divide.

It's been fun sparring with you :)

all the best, eli