Refine the validator memory optimization so that only directly created
validators are nulled out. Also update the logic to ensure that
`destroy` is also called.

See gh-4734