Infants in sub-Saharan Africa are exposed to diseases and undernutrition which can have long-lasting consequences for their health and education. In this paper, an indicator for adverse environment in infancy is created and its association with height-for-age and school attendance explored. Further, the role of household living standards in sheltering children from adversity is considered. The results show that adverse environment in infancy negatively affects height-for-age later in childhood, however only at the lower end of adversity distribution. The association with education is mixed. At the relatively low levels of adversity, a negative association with school attendance is observed while a positive association is observed at high levels of adversity. Children in better-off household appear to be more affected.