This is where your logical argument breaks down. Yes anarcho-libertarianism is against state power, but that doesn't validate the government's role as referee.

You are assuming the distribution of power is a zero sum game where there are winners and losers. It is governments job to provide scarce resources and services equally, or to those in greater need.

What if you eliminate the 'scarcity' mindset and instead had an 'abundance' mindset? What if everyone got everything they needed in life? This would eliminate the need for government.

Jacque Fresco believed in computerized central planning where computers ( now AI ) gave humanity everything it needed. No need for private property. No need for money. No need for government. Although this is just a thought exercise it shows it is only because of scarcity of resources that government has power over us. It proves that government deliberately creates scarcity so people will willingly become 'subjects'. The promise of modernity was that technology would eliminate scarcity and we could live in a utopian world.

Government is one reason we don't have free housing, healthcare, education, and other basic needs. It doesn't act as a referee, it acts as a protection racket for the rich and powerful.