The stresses of everyday life may start taking a toll on the brain in relatively early middle age, new research shows. A study of more than 2,000 people, most of them in their 40s, found those with the highest levels of the stress-related hormone cortisol performed worse on tests of memory, organization, visual perception and attention.

Higher cortisol levels, measured in subjects’ blood, were also found to be associated with physical changes in the brain that are often seen as precursors to Alzheimer’s disease and other forms of dementia, according to the study published Wednesday in Neurology magazine.

The link between high cortisol levels and low performance was particularly strong for women, the study found. But it remains unclear whether women in midlife are under more stress than men or simply more likely to have their stress manifested in higher cortisol levels, says lead researcher Sudha Seshadri. A professor of neurology,  at both the  Boston University and The University of Texas Health Science Center at San Antonio, where she is the founding director of the Glenn Biggs Institute for Alzheimer’s & Neurodegenerative Diseases.

Working on the study “made me more stressed about not being less stressed,” Seshadri says, laughing. But, she adds, the bottom line is serious: “An important message to myself and others is that when challenges come our way, getting frustrated is very counterproductive—not just to achieving our aims but perhaps to our capacity to be productive.”

The new research included volunteers from the Framingham Heart Study, a 70-year-old study of residents from a Boston suburb where most of the the participants, were white, middle class and suburban, Yet even these relatively young and apparently well-off people showed signs of brain changes, both in brain scans and in their performance. “This is the range of stress that a group of average Americans would experience,” Seshadri says. The highest cortisol levels were associated with changes that could be seen on an MRI scan of the brain, the study found.

Earlier research has shown weaker-than-average performances on tests like these are associated with a higher risk of dementia decades later, and Seshadri says high stress levels in midlife might be one of many factors that contribute to dementia. Understanding that link might offer a potential opportunity to reduce risk—but she cautions research has not yet shown conclusively that lowering cortisol levels will reduce the risk of Alzheimer’s.

Other research has shown cortisol levels can be reduced with adequate sleep, exercise, socializing and relaxing mental activities such as meditation.  Rockefeller University’s Bruce McEwen, who studies how hormones act on the brain, says it is never too late to adopt a healthier lifestyle by taking steps like reducing stress, exercising regularly, eating a healthy diet, getting enough good-quality sleep and finding meaning in one’s life. “The life course is a one-way street,” he says. But “the brain does have the capacity for repairing.”