My take on this is two-fold:
Firstly the problem with B&O equipment is that it can't be interchanged with third-party equipment, which is something that a lot of people like to do. Connect B&W speakers to a Primare AV unit and so on. At least this is the perceived perception - that you need to buy a B&O hifi, speakers and so on, and this can prove to be expensive and audiophiles might think that you've copped out of making a hard decision, by buying the kit from one source.
The other problem is the type of people that B&O stores traditionally attract. Our local store quite rightly say that the majority of B&O customers are seeking quality, ease-of-use, support and installation over state-of-the-art, latest features, competitive pricing etc. The typical B&O customer seems to be 40 upwards, has a high disposable income, has no interest in looking around for the best deal and simply wants to walk in to the store, make a decision, then get it installed. For this reason, so-called experts or technophiles are wary of B&O kit as they see the type of customer attracted to B&O kit - I feel the same way. Many a time I've been in to my local store to see a couple in their 50s, with £10K to spend on a TV and speaker combination, getting a demo of the various B&O TVs. It's very rare to see someone <35 in the store, making a purchase. Indeed, if I'm ever in a B&O store looking at kit with another person <30, they often feel overwhelmed by the pricing to the point they do not return for a long time.
It's weird. Some people would never dream of buying B&O kit. My brother is one of these people. Both him and his GF are professional architects, sitting in a superb modern self-designed house. For some reason, when I asked him about the tech kit he was planning for his new house, B&O kit was now way down on the list. He has been in to a local B&O store, but wasn't overly impressed with the picture quality - he loves the design of the Beovision 8, and the Beocenter 2 and was tempted to buy, until he was told how much the Beocenter 2 was - add the two together and it is a £5600 purchase for a DVD player that connects via scart (!) to a low-end TV that's made in China. To be fair, you'd have to be absolutely flush with cash to justify that decision.