As a matter of fact libertarians believe in something more like "do Whatever you want as long as it doesn't involve violence" and "self-defense isn't violence"