When speaking negatively about someone, whether about an Aveirah that person did, or a bad middah he has, one transgresses the aveirah of Lashon Hara. The same is true if a negative remark can cause some type of loss to the person spoken about. This is described at length in the Sefer Chafetz Chaim.
I am looking for the source (in the Chofetz Chaim) that speaking other negative things about someone, not related to an aveirah or middah and which cannot cause harm, is also not allowed because of Lashon Hara.
For example, is telling a humorous story about someone who slipped on a banana peel Lashon Hara? It is neither an Aveirah nor a middah.