You lose control of the creature if the Old Man leaves the battlefield. This is because it is no longer tapped.

If Old Man of the Sea stops being tapped before its ability resolves -- even if it becomes tapped again right away -- you won’t gain control of the targeted creature at all. The same is true if the targeted creature’s power becomes greater than Old Man of the Sea’s power before the ability resolves.

The ability doesn’t care whether Old Man of the Sea remains under your control, only that Old Man of the Sea remains tapped and its power remains large enough. If an opponent gains control of Old Man of the Sea, you’ll keep control of the affected creature (until Old Man of the Sea stops being tapped or it no longer has enough power).

Once the ability resolves, it doesn’t care whether the targeted permanent remains a legal target for the ability, only that it’s on the battlefield and its power remains small enough. The effect will continue to apply to it even if it gains shroud, stops being a creature, or would no longer be a legal target for any other reason. If it stops being a creature, its power is considered to be 0.