Well, nothing. Everyone has different balances of desire and empathy.
In my country there's a huge deal about rhino poaching for their horns. Pictures of dead rhinos with only their horns chopped off generally horrify us and most can't imagine why someone would kill a rhino for its horn. However, I recently read something about how these pictures have no impact on foreign purchasers of rhino horn - the desire for the commodity far outweighs any feeling for the animal, despite it being endangered.
So you might care about the rhino, but not about your food. Some might care about orphans, but not rhinos or food. Others might care about everything but vegetables. Who is correct? I don't know, but I think it's important to at least engage with the issue so our decisions are conscious. The end result isn't as important as simply having gone through the process.
That was not GP's argument and it certainly wasn't a good reply. GP simply effectively said, "The circumstances are irrelevant to me, it makes me feel good so I'll do it." Your argument is you have an intimate understanding of where you food comes from and you're okay with that.
I sometimes draw an analogy between veganism and the extreme anti-force / volitionalist stance of radical libertarians. In both cases is may be radically moral, but the problem with the libertarian case is similar to what you just said.
Government force is just so easy. Is there some systemic social ill that manifests in an emergent fashion from the market? Pass a law and force it to change. Wealth redistribution by force improves the lives of the vast majority of people too.
There may be force-free alternatives to these mechanisms, but it's clear that if they exist they are more difficult. Thus most people are willing to sacrifice some of their (and their neighbors) freedom and autonomy for free health care, free retirement, easy fixes to emergent problems and tragedies of the commons, etc.
I can't say I'm different. I would prefer to have a "softened" market in which some wealth is redistributed by force to me and my middle class contemporaries (as well as to those less fortunate than me) even if it is less moral according to a radical moral theory.
I'm glad to finally see someone who supports wealth redistribution clear that it requires force (as well as being in their own self-interest). While I can respect this widely held position I am personally uncomfortable with this aspect. I wish we could achieve the goals of redistribution (btw-these goals need to be clarified) without force (i.e. voluntary charity). This actually seems possible to me because of the strong interest in charity by so many especially the very wealthy and the incredible inefficiency in the delivery of aid to the poor.