The incentives are completely screwed, because flaring off that gas represents a mere opportunity cost to the oil company, yet imposes a heavy externality on the rest of the world. The waste is only "less expensive than the alternative" because the person doing the wasting isn't the one paying the price.
So yes, absolutely: impose a carbon tax, where the person burning any fossil fuel pays a price that actually reflects the externalities of their choice. That's not arbitrary at all. If it still makes financial sense to flare rather than capture, fine. But there's no pretending that everything is all right with the existing equation.
But isn't the point that the externality is in fact not that heavy? That implies that the process is at most 97% efficient, which is actually pretty damn good (of course, there'll be other inefficiencies, but nobody's complaining about those). If the rest were burnt in internal combustion engines to move cars about, it's almost certainly < 30% efficient and very often < 20%, which implies most of the externality comes from people further down the chain.
Just to make sure we're on the same page: I'm talking about the externality of global warming, which at present is completely absent from the oil pricing system.
As a result, I'm not sure it's as simple as a straightforward efficiency calculation, because under a system where the consumer bears the currently externalised costs of burning fossil fuels, both the cost of flaring would go up, and the oil value captured by the drilling company would go down. I have no idea what a sane level for a carbon tax would be, and I'm certainly not economist enough to predict the ramifications for global fuels pricing; like I say, it may well work out that flaring remains cost-effective. But the costs of an action should be borne by the party that benefits, otherwise the incentives are, as I say, totally screwed.
I believe the implication is that the environmental impact of building natural gas pipelines criss-crossing the U.S. so we don't have to see flares in North Dakota may outweigh the marginal impact of the CO2.
So yes, absolutely: impose a carbon tax, where the person burning any fossil fuel pays a price that actually reflects the externalities of their choice. That's not arbitrary at all. If it still makes financial sense to flare rather than capture, fine. But there's no pretending that everything is all right with the existing equation.