How Many is Too Many?

I am a People magazine junkie. Since I no longer subscribe to the magazine (too expensive), I visit the website at least once a day for my fix. This week, the site had a story on the 19th Duggar baby leaving the hospital. I know I should have avoided the reader comments, but I went to that section anyway. I'm stunned by the things people had to say. Yes, I know the Duggars are a controversial family, but so many people not only insisted the family needs to stop having children but also blamed the baby's prematurity (25 weeks, I think) on the fact that she was their 19th child. Some people basically said the family deserved to have a fragile preemie.
I don't watch the Duggar's show. It doesn't appeal to me, mainly because I have enough chaos in my own house. From what I hear from people who do watch the show, the family seems to be a loving one and the children are fairly well adjusted. Duggars aside, I wonder this: why do we feel it is our place to tell anyone how many children they should have.
When the subject of more children comes up, people often say "you're done having kids, right?" Why? In theory I get it: In this day and age, four children is considered a large number, four at once can be unthinkable. But it's not unheard of for families to have multiples and then decide to have more children, nor is it irresponsible. Having a large family doesn't automatically equal living on the dole.
I never imagined myself having more than two children. Now here I am with twice that number and I love my life. Families come in all sizes. We walk a slippery slope when we put limits on what that size should be.