The transaction is for about €32.5m (£24.5m) in cash and €12.5m (£9.45m) in an interest-free loan note due in 2013.
Warnaco said it expected the sale to be concluded during the first quarter of 2008.
Warnaco group president of intimate apparel and swimwear Helen McCluskey said: "Under Palmer's ownership, Lejaby will continue to be operated from Lyon, providing continuity to its core employees, and benefit from new opportunities. Today's announcement completes the strategic realignment we announced in November, including the company's exit from all owned manufacturing. We believe Warnaco is now well positioned to focus our efforts on our brands and businesses with the greatest long-term potential for Warnaco and its stakeholders."
Palmers chief executive Thomas Weber said: "We are thrilled by the acquisition of the Lejaby business. We believe adding Lejaby, an iconic and well established brand, to our portfolio will surely enhance our future growth opportunities."