I'm in Trouble Because the Emperor Thinks My Time Is Limited - Chapter 43

Based on little snippets into his memories it seems it was because he was abandoned as a young even though as part of a herd he should have been taken in if the parent was gone