If you peruse any given SE site user's accounts page (mine included) there's a good chance you'll find that they are a member of a site in which they don't actively participate. People often join sites out of curiosity or over-zealousness, but come to realize they really don't have anything to offer that community. However, someday they might.
Currently, as far as I know, the only way to get out of a site is to ask to have your account deleted. It would be nice if we could deactivate our own accounts. This would remove them from the account list when others view your profile (perhaps even hide it from the owner also, in a collapsed region or something), thus reducing clutter. Should we decide one day that we're ready to actively participate, we can reactivate the account and continue where we left off. If the user has ever participated on that site, I think the account should still appear in the user list, but there would be some indication that the user is "on hiatus". If they never participated, there's no reason for them to show up in that's site's user list.