Clothing spending contracted 2.9% in November, the biggest drop since October 2017, data from Barclaycard shows.
Department stores were particularly hard hit, declining 7.1%.
Overall spend last month showed the lowest year-on-year growth since March, up just 3.3%. Growth came from increased fuel prices, ticket sales and spend at pubs and restaurants.
Only 52% of UK adults said that they have confidence in their household finances, the lowest result since 2015, and 64% of shoppers say they are being careful to seek out value for money in the purchases they make.
There is some hope for a good December, though, as only 16% said they expected to spend less in December as a result of shopping at the end of November.
Esme Harwood, director at Barclaycard, said: “November was a mixed month, with department store and clothing spending contracting but entertainment remaining strong. It seems shoppers are yet to make their main Christmas purchases for friends, family and loved ones – despite many retailers offering Black Friday discounts to try and boost sales.”