Look at their Likes and Dislikes. You can (usually) house dinosaurs that Like each other with no problem, and it doesn't add anything to the Cohabitation meter. There are some dinosaurs that Like another species that is Neutral to them, so the one that likes will remain at 0% Cohabitation while the other gets 25% or so on their meter. With a bit of planning and matching you can have up to four different species together without a fuss. The compatibilities of every species is different, and many have a Like of only one particular species of a group (example, Edmontosaurus only likes Archaeornithomimus and not all three ornithomimids). All armored dinosaurs dislike other members of their group (ceratopsians don't like each other, neither do stegosaurids and ankylosaurids) and generally ceratopsians dislike stegosaurids and vice versa, though they both like or are neutral towards anktlosaurids.
Theropods are a bit trickier. They do have likes and dislikes of other theropods, and those that like each other will cohabitate fine, at least for a while. But for whatever reason, at least with the actual dinosaurs, they'll come to blows no matter how comfortable they are and injure/kill each other at some point. I haven't played with mixing herbivores and carnivores yet, but I believe that for the ones that are neutral to each other, you can have one or two theropods of the same species in an enclosure with herbivores and if they have enough of their needs and the enclosure is big they'll leave the herbivores alone.