Victoria has been a provincial backwater for most of its history. Despite being in a mild winter setting, it's so small when compared to Edmonton, Winnipeg, Quebec City and Halifax.
https://www.onevictoriaplace.ca
https://skyscraperpage.com/diagrams/?cityID=37&status=15
While Edmonton was eventually allowed to have a tall building, even by Toronto, Calgary and Montreal standards, Victoria was always supposed to have shorter buildings than Winnipeg, Quebec City and Halifax. That's part of the KEEP THINGS SMALL mentality on V. Island.
However, as more people discover that Victoria and Vancouver are the mildest winter cities in Canada, more people just might want to move there. Especially, when Canadian Snowbirds don't feel as comfortable with Florida, Texas & California.