Is there a way to make a question or answer I post not visible in my profile?
You can't make a question or an answer private AND keep it linked to your account. We have no plans to implement anything like that at the moment.
However, we can anonymize the posts for you. To do that, send an email to the "contact us" link at the bottom of any page and give us links to the questions you want disassociated from your account.
Your name will be removed from the post and it won't be linked from your profile anymore. If anyone mentioned your name in comments, that'll remain, cause we don't really keep track of that sort of thing.
(And of course, as others have said, be careful when posting sensitive information in the first place.)
You could delete the question or the answer and it would no longer be visible to anyone except 10k+.
Unless you really no longer want this to be available and or helpful to anyone, then I don't know why you would want to do that.
EDIT: Or as @BenBrocka and @animuson stated in the comments you can email the team to either disassociate the site with your main account or flag the post to disassociate the content with your profile.