If Vexing Sphinx has more age counters on it than you have cards in your hand, you can't pay the upkeep and you must sacrifice it.

Paying cumulative upkeep is always optional. If it's not paid, the permanent with cumulative upkeep is sacrificed. Partial payments of the total cumulative upkeep cost can't be made. For example, if a permanent with "cumulative upkeep " has three age counters on it when its cumulative upkeep ability triggers, it gets another age counter and then its controller chooses to either pay or sacrifice the permanent.