You are perpetuating the stigma we face, and you are talking about it like you are not part of the problem, worst, you are blaming us for facing said stigma by being parents and wanting to build a regular life for ourselves. The stigma does not come from others, it comes from you when you say we should not have children, it comes from our own clients who think we don't deserve what everybody has because we are sex workers, it comes from each of you in this society who thinks we do not deserve to have regular lives.
Breaking the stigma starts with you, by believing that we deserve to be parents, by believing that we can be integral parts of this society, and educating people around you and spreading the word that we deserve respect. If everyone would observe and question their prejudices against us (i.e. it starts by realizing that statements such as sex workers shouldn't be parents ARE stigmatizing us), we would be able to raise children normally, without facing the stigma you so beautifully acknowledge, and yet, perpetuate.