When I read that sentence, I thought you were being satirical. It is utterly nonsensical to assume any given number is "the only sensible number to have".
I prefer a smaller guild, and when my guild had around 120 members with 40 or so on at primetime, many found it to be pushing the limits of what they enjoyed; too much chatter, too much noise. Sure, you can turn guild chat off, but many prefer to have a small idle chat going on now and then and a more tight-knit community. Thankfully, most of my guild got bored of guild wars a few months after we hit that point, so the guild promptly died and the day was saved.
I also enjoyed knowing most of my members, rather than just seeing "Oh, you're in the guild, too" and one thing I didn't like about the guild growing so fast when it did was that there were tons of people I didn't know. I wasn't complaining, it was just unusual for someone who had led a <20 guild for five years prior.