Calderwood lost his job as Dons boss in May 2009 despite taking the club to the last 32 of the UEFA Cup, securing another season of European football, and always achieving top-six status in the SPL in five years at the helm.
The Dons have been in the bottom half for the bulk of the subsequent period with Craig Brown and Archie Knox struggling to improve the poor results under predecessor Mark McGhee.
Second-bottom Aberdeen go into Friday night's visit of Dunfermline having failed to score in seven of their nine SPL games this season and Calderwood admits he expected better of the team.
Calderwood, speaking at the second-round draw for the William Hill Scottish Cup, said: "They are very experienced, him and Archie, and it's always the manager who gets the blame.
"I've not seen them but looking from the outside, which is always dangerous, I don't think the quality of player they have brought in are as good as the quality of the ones who have left.
"A lot of experienced players, and winners, have left the club. Zander Diamond left, Derek Young knew what Aberdeen was about, Mark Kerr, Gary McDonald, good pros who know what's needed to win a football game.
"But it might be luck, I haven't seen them this season."
Calderwood added: "It's a massive game for them now on Friday. They have Dundee United, Rangers and Celtic after that.
"The expectation would be top six at least and probably higher.
"If they do lose, it could be very difficult.
"It's a worry and it's a worry for Scottish football also. Aberdeen only took 450 to Easter Road (against Hibs), which is one of the bigger games of the stadium.
"So it's costing everybody and Scottish football needs a good Aberdeen team."
Calderwood appeared to pay the price for the high expectations at Pittodrie after failing to achieve cup success, but he does not believe the club's past glory under Sir Alex Ferguson weigh down those now at the club.
"I had that in the beginning but not in the end," he said.
"But they do expect to get in finals and be high up in the league.
"And financially-wise, even though everyone is struggling, they are probably the fourth biggest team in Scotland.
"I think they should be doing better than they are but it's easy to blame the manager all the time."
Calderwood does not feel vindicated by Aberdeen's struggles and still does not understand fully why the board decided to move on.
The former Dunfermline manager said: "I don't think I had anything to prove. I thought we had a magnificent record - three times getting into Europe with two different clubs.
"Okay we lost in the cup semi-finals but every year in the top six, I think that says enough. But I'm still not working."
He added: "It could be character, some people like you and some don't.
"That's the things that annoys me, I always had my own choices in life when I was leaving a club.
"That wasn't my own choice and I didn't see it coming, which really annoys me, because I'm usually quite clever when it comes to those things. But I wasn't then."
When asked if he would ever return to Pittodrie if asked to go back at any stage, he said: "I don't think I would be asked.
"It broke my heart leaving, I love the club, I love the city.
"I would need to talk to a few people first and get a few truths from people."
Source: Team Talk
Source: Team Talk