A few years ago I had the worst possible intern ever. I had just joined an organisation and was excited to work together with her. I was determined to make her time in the organisation and in the country a success. For her it was, for me it was hell on heart. 

When she left I thought "What if I am asked for a reference? Would I be ok in telling people how bad she was? Would I ruin her future chances?"

The best choice is to warn people in advance so that they won't mention you with potential employers - which I didn't, she left in hurry and refused an exit interview; but there is always the chance you are reached by recruiters or potential employers. And that's what made me question my intentions.

Some managers just refuse to comment, to avoid any possibility of lawsuit, but they are not making a courtesy to their peers. And it would sound suspicious anyway. You may be willing to not jeopardize people's chances for a new job, but providing a false positive reference will surely jeopardize your reputation.

Giving bad references does not equal defaming. As long as what you are saying is factually accurate, you will be fine.