In 2004, reports emerged that Avril Lavigne and Duff were having a spat. Newsweek quoted Lavigne as saying, “I read that I was supposedly mad at my fans for dressing like me”. Lavigne further added,

They quoted Hilary Duff saying, ‘Avril needs to appreciate her fans more and blah, blah, blah.’ I’m like, excuse me? First off, it’s not even true. I never said that. And second, who the hell cares what she has to say about my fans? Whatever. Hilary Duff’s such a goody-goody, such a mommy’s girl”.

In June 2007, there were reports of a run-in between the two at a Maxim party in New York. Evidently, a complication in the arrival time of both caused the unwanted run-in. Lavigne was said to be furious at this, pacing around and pressuring her company to leave.

In June 2006, when asked about her then-boyfriend Joel Madden, a native of Waldorf, Maryland, in an interview with Elle, Duff replied, “He’s very real, like, he’s from a pretty ghetto place in Maryland… I like that”. Duff’s “ghetto” comments sparked a mixture of mild offense and bemusement from residents of the Waldorf area, as the suburban, mostly middle-class town did not fall into the traditional impoverished inner city concept of a ghetto.[93] Duff later claimed she was referring not to Waldorf, but actually a section of Baltimore where Madden lived briefly as a young adult.

In late 2006, Duff took legal action against an alleged stalker and his roommate. On November 3, 2006, Duff’s stalker, Maksim Miakovsky, was arrested for threatening to kill her. He was booked on charges of making criminal threats and stalking. According to legal papers filed by Duff, Miakovsky came to the U.S. “for the sole purpose of meeting and becoming romantically involved with Ms. Duff”. Miakovsky was being held on $200,000 bail at the Manhattan Beach jail. He was arraigned on November 7, 2006. On January 19, 2007, Miakovsky was sentenced to 117 days in jail and five years probation after pleading no contest to the charges.

By 2005, Duff appeared to have lost weight, leading the media to speculate that she had developed an eating disorder, although Duff denied this claim. Duff was interviewed on the Australian current affair show Today Tonight and stated that she lost weight by leading a more active lifestyle. Later in 2006, an article on Digital Spy stated that Duff slimmed down to a U.S. size zero because of media reports suggesting she had gained fifteen pounds. She later stated that she had been feeling the pressure to be thin, because she is perceived in the media to be either too fat or too thin. Duff called this “judgmental” and “mean”.

In January 2008, videos of Duff surfaced on YouTube, which showed her singing into her microphone at her concert in Mexico, but her voice was inaudible, leading people to believe that she was lip-syncing. Duff’s representative spoke out about the incident saying, “She was not lip-synching. It was faulty equipment. There was no sound coming out, but she was singing”. He also added that Duff was not using her regular sound equipment, but instead was using locally provided equipment and that her microphone was inadvertently set on mute for the first few minutes of the performance.