His departure follows the completion of a refinancing deal for SRG, by private equity owners Gresham earlier this month.
The Brick family sold the business to Gresham in 2005 in a buyout led by the former Moss Bros chief executive Adrian Wright. Former managing director Brian Brick remained involved as non-executive director.
SRG sold its Chester Barrie business to Japanese manufacturer Itochu earlier this year and is selling its Woodhouse stores to fashion agency Four Marketing.
SRG also owns Young's Hire and Racing Green.