Couple of points to thi k about, that may or may not sway your story in some way....
- not all imperial slaves are treated well and legally. Slaves in 'edge' systems away from the centre of the empire, where influence and security are less, are probably treated more like regular slaves (mentioned in Elite: Reclamation).
- people may have become slaves while pregnant or while they have young children. I wouldn't expect the children to be taken off them - they would raise their children along side their slave duties. Some slaves may even get pregnant while in slavery (a couple may enter slavery together, or their slave 'master' might be the father).
I think alot of it depends on where they are located, what their 'master' is like, etc and IMO, you wouldn't be going against lore by being a child raised in imperial slavery. My back story is similar, but coming from regular slavery, not the imperial side of things.