If the creature targeted by Ancestral Vengeance’s enchant creature ability is an illegal target as Ancestral Vengeance tries to resolve, Ancestral Vengeance will be countered. It won’t enter the battlefield, and its enters-the-battlefield ability won’t trigger.

If Ancestral Vengeance causes the enchanted creature to have toughness 0, that creature and Ancestral Vengeance will be put into their owners’ graveyards. The enters-the-battlefield ability will still apply.