Vengeful Rebirth has two targets: a card in your graveyard, and a creature or player.

You can return a land card to your hand with Vengeful Rebirth. It just won’t deal any damage to the targeted creature or player if you do.

If the targeted creature or player becomes an illegal target but the targeted card in your graveyard doesn’t, Vengeful Rebirth will still return that card from your graveyard to your hand, but it won’t deal any damage. Vengeful Rebirth will then be exiled.

If the targeted card in your graveyard becomes an illegal target but the targeted creature or player doesn’t, Vengeful Rebirth won’t return that card to your hand and it won’t deal any damage (because you didn’t return a nonland card to your hand this way). Vengeful Rebirth will still be exiled, though.

If both targets are illegal, Vengeful Rebirth is countered. It will go to your graveyard.