I think the "don't have a worldcon, it destroyed Victorian fandom" concept came from the misconceptions of Perth fans who mostly had close to nothing to do with Melbourne fandom before A3. Those misconceptions were further fueled by reports of how A3 was run, which were largely the fault of A3 in particular, not Worldcons in general. Unfortunately, those who have only seen one example of something will tend to assume that it is typical.