Size definetly matters. Trying to breed 4x100 red pandas atm (managed to get two so far), so all my breeding pairs are at 92-100 in both size and longevity and all of them gets five stars befor they reach 15 years old, some even befor they are 10.
But the star system is really strange when it comes to some animals. At the last weeks challenge, I had loads of Elephant babys and I had to put the males in the trade centre as soon as they grew up to avoid fighting. When I put them in the trade centre, all of them were at one or two stars. But when I put them back out in a habitat, some of them were still at 1-2, but many of them were for some reason at 4-5 stars?? I've seen loads of Elephant bulls at 5 stars as soon as they mature (never directly in the habitat, but either when putting them in the tradecentre and then back out or buying young bulls on the market). So ye, the starsystem is pretty wierd sometimes...