Little is known about the relationship between maternal affectivity, social withdrawal and temperament in infants in low-income countries. The goal of the study was to assess the prevalence of social withdrawal behavior in infants aged 8 ± 2.3 months and to explore associations between maternal affectivity during pregnancy and postpartum, infant social withdrawal (as a sign of stress) and ‘difficult’ temperament as assessed by the mothers. 458 mother–infant dyads were recruited in the city’s public mother and child health-care centers. The eight items of the Alarm Distress Baby scale (8-ADBB) and the five-item M (modified) ADBB (M-ADBB) were used to assess sustained withdrawal behavior (ISSWB). The Goldberg Depression and Anxiety Scales were used to assess maternal affectivity and mental well-being. A specially designed questionnaire was used to identify stressful events faced by the mother during pregnancy. The ELDEQ-QCB was used to assess the degree of difficulty in managing the baby. Using the M-ADBB, we found a striking figure of 69.2% for ISSWB with 8-ABB (range 0–29) and 72.7% with the M-ADBB (range 0–10). ISSWB was linked to negative maternal affectivity and to high incidence of stressful events for the mothers, and to the child being viewed as ‘difficult’ by the mother. Positive prenatal affectivity was a protective factor of ISSWB (OR 0.46). Results are compared with previous studies in Africa. Early screening for ISSWB and identification of factors affecting maternal mental well-being could help in early intervention and increase the chances of better child development.