I've had similar thoughts but you've put them into words better than I have. Basically, I think a new ancap society could only exist with people that chose to be a part of it, it can't have anyone there unwillingly and even would need an outlet for people to leave if they couldn't take it. In the absence of this I feel it would devolve into constant violence as many people simply aren't capable of fully taking care of themselves and they would cause a lot of problems for the self sufficient people.