Per-site close reasons were created to reduce the amount of confusion and debate surrounding what specifically is on- or off-topic for a given site. Folks asking questions for the first time are often unclear on what is allowed or expected; by explaining common misconceptions in a clear manner, you can help these people better understand and learn to become productive members of your community.
You need to be aware of what's on- and off-topic for your site
If you've already had multiple discussions about what should go into your site's on-topic and about pages, then you probably have a reasonable idea of where the boundaries sit here. If your community has worked to provide a solid set of FAQs on your meta site, then you're in great shape. But if you're still relying on the default "audience description" that came from Area51, you might have some work to do...
You need to know what the most common misconceptions are
You can find a list of recently-closed questions in the 10K tools. I've found it useful to look at a random sampling of recent questions closed as off-topic and too-localized. You can do the same using this query... but be aware that it excludes deleted posts, so the results will be somewhat skewed; also, beta sites aren't in SEDE. If you want a list for your site, ping me in chat.
Don't try to brainstorm on what you think should be or might be off-topic. Critically examine what you're seeing in practice instead - this should be driven by the community, not by your own personal preferences or prejudices.
Discuss it with your community
Closing should be something that has wide-spread support on a site. If folks disagree as to what is on- or off-topic, then that's something that needs to be resolved before going further. In particular, topics that are allowed with restrictions need to have those restrictions hashed out and documented, either on your meta or on your on-topic page.
So take your ideas for off-topic reasons, throw them into a meta post and let folks discuss them.
Create the wording for each reason as it will appear on the site
This can be tricky; your goal here is to communicate clearly with the people whose questions are being closed, with the people who are closing questions and with the people who will later read those questions and wonder why they were closed.
At a bare minimum, off-topic reasons should identify a specific topic considered inappropriate. If you're struggling to be specific, find an example of a question that is off-topic and discuss the factors that make it inappropriate for the site.
Whenever possible, try to explain why a given topic is not allowed - this is your chance to answer the inevitable question once rather than repeating it every time a question must be closed.
Provide resources that will aid askers in solving their problems: either instructions for asking the question in a more suitable fashion, links to a different site where the question may be considered on-topic.
Be as concise as possible. Save lengthy explanations and examples for meta FAQs that can be discussed and refined over time, and then include links to these in the community-specific close reasons. Don't depend on anyone following these links though; keep critical information and guidance in the reason text itself.
Do all of this on your meta site so that folks can critique the wording and suggest improvements.
Pick the top three reasons
Large lists make selection tedious and difficult. Fortunately, most sites do not attract an extremely high volume of off-topic questions, and those that are asked tend to be about a relatively small number of topics - therefore, we limit the number of active reasons to 3. A few sites do get a regular influx of off-topic questions in a larger number of distinct categories; if you find this to be the case on your site, post a feature-request on your meta site with examples of questions that would benefit from a predefined community-specific close reason option - we'll verify the need for this and work with you to make more reasons available.
Add the reasons, approve them and activate them
Any moderator can add a new community-specific close reason, and up to 3 custom close reasons can be active at any given time.
Once you've settled on the wording for a reason, a moderator will need to approve it.
The moderator who created a reason cannot approve it; we want at least two people to be reviewing these before making them available. Approving a reason also activates it, and as noted above, only 3 reasons can be active at any one time on most sites - to approve more than this, an existing reason will have to be deactivated first.
Community-specific close reasons can never be completely removed from the system - this prevents reasons that have been used on closed questions from disappearing and causing confusion. However, moderators can deactivate reasons at any time, which will prevent them from being used in the future (unless re-activated).
You can also drag active reasons to change the order in which they appear on the close dialog.
Monitor the use of your community-specific close reasons
A tool for monitoring the usage of these reasons is available to moderators and 10K users on every site.
It's important to keep an eye on how the community-specific close reasons are being used on a site. If good questions are being closed incorrectly, the predefined reasons may be too vague; if too many bad questions are being left open or closed with "other" (custom comments left by the close-voter), then you may have picked the wrong reasons. If you see something amiss, don't hesitate to bring it up for discussion on your meta.