I've been researching this for mother in law who has a neighbourhood cat using her yard as a sand tray. It also has been hunting baby brushtail possums, which is a BIG no-no. mother in law really hates finding dismembered corpses of her favourite friends.
What I've found -
1) sprinkle pepper
2) sprinkle/spray vinegar/water mix (strong enough to smell vinegar, doesn't need to be too strong)
3) use lemon or citrus oils instead of the vinegar; or citronella oil
4) Buy a commercial cat repellent/cat deterrent (which are mostly based on citronella oil)
And some tips from another website :
"Cat repellent sprays and gels are available from many plant nurseries, pet stores, and some hardware stores.* You can also try making up your own repellent using a number of scents that cats dislike.* We suggest trying the following:-*
A mix of ground black pepper & chili
Scatter citrus peel (oranges & lemons) around the garden
Place pine cones around near garden borders
Plant geraniums, marigolds and petunias, cats apparently dislike those plants
Grow spiky plants near areas you want to keep cats from (base of trees that attract birds)"
And from another site:
"* amonia soaked (corncobs, etc)
* aluminum foil
* bamboo skewers
* black pepper
* blood meal fertilizer
* bramble cuttings
* Carefresh - "recycled" wood pulp
* catnip - donated into your neighbor's yards (so they'll stay in their own yards)
* cedar compost
* chicken wire (metal or plastic) [husband says be wary - metal chicken wire can poison your soil, the zinc in galvanised wire is not a good idea in the garden]
* cinnamon
* citrus peels
* citrus spray
* cocoa bean shells
* coffee grounds -fresh & unbrewed, not just a light sprinkling (highly recommended by MANY Gardenwebbers!)
* dogs
* electric fence for animals
* essence of orange. essence of lemon, lime (citrus essential oils)
* fresh manure(ditto)
* garlic cloves
* gumballs from the Sweet Gum Tree
* gutter covers
* hardware cloth
* heavy bark mulch
* holly leaves
* keep the area damp, they like dry soil
* lavender
* liquid manure (good for your garden too)
* motion sensor sprinkler
* pennyroyal
* pinecones
* pipe tobacco
* plastic forks
* predator urine
* red wine vinegar
* river rocks over the exposed soil
* rocks, crushed
* rose bush clippings
* rue, an herb (Ruta graveolens) (highly recommended in plant form only)
* short twigs throughout the planted area about 6" apart
* six-inch bamboo skewers (pointy side up)
* Spray on your leaves (not the cat): fill a spray bottle with 1/2 t chili powder, 1/2 t cayenne pepper, 1 t dish soap and water
* squirt gun with water
* talk to your neighbors
* tansy
* thorny berry, lilac, hawthorn, rose clippings
* toothpicks
* upside down vinyl carpet
* vinegar sprayed on areas where they roam
* water bottle on "stream"
NOT RECOMMENDED:
*** chili powder, red crushed pepper, cayenne pepper (NOT recommended), it gets on the cat's paws then they wash themselves and they get it in their eyes, beware cats have literally scratched their eyes out because of this. Even if it's one cat out of 500 infected in this way, that's one too many for me.
*** Don't ever use mothballs or flakes. Those little toxic waste pellets destroy cats' kidney function, could seriously harm people who handle them, and yes, contaminate your own garden soil. Their packaging even warns against using them this way."
I hope this helps. I really like the holly leaves idea - all round the property where the cat is getting in, and especially round the window where the cat is spraying.
Marg