The Serbian world number one made light work of his French opponent on the clay in the first set but struggled in the second.
Djokovic saved set point at 4-5 with a fine serve, preserving his record of never having dropped a set against Chardy in 13 matches.
He went on to seal the tie-break with a forehand winner.
“I was set point down. It could have gone a different way,” said Djokovic, who won in Madrid in 2011 and 2016.
“I wasn’t really thinking about my score against him in that moment. I just wanted to play it well.
“The wind picked up in the second set. So, it was faster and you had to just pick a position sometimes and kind of rely on your instinct, and it worked well.”
Djokovic will face Marin Cilic on Friday, after the Croatian beat Laslo Djere 4-6 6-3 6-2.
Dominic Thiem also progressed with a 6-4 7-5 win over Fabio Fognini.
In the women’s tournament, Simona Halep reached the semi-finals with a 7-5 7-5 win over Ashleigh Barty.
Two-times Madrid Open winner Halep, the third seed, won consecutive points with down-the-line forehands to finish off her Australian opponent.