TBH even during Voldemort’s era I don’t think many normal folks were impacted except if they were in the wrong place at the wrong time or briefly when they took over the Ministry. So it’d of been pretty minor overall for your average wizard I think (correct me if I’m wrong, been a while since I read the books).