So it’s made of shit, right? And shit is an animal product. But barring a night of drinking or a particularly aggressive burrito, shitting does not harm the shitter; it’s beneficial and required. Also the animals in question can and do consent, does that make it vegan?
Modern fertilizer is generally made through chemical processes. It’s why we don’t harvest guano islands anymore. Nowadays they use the haber bosch process to make ammonia from atmospheric nitrogen.