The actors - who play Thor and Captain America respectively in the Marvel Comics film - said that the experienced 47-year-old spearheaded a mainly younger ensemble cast including Scarlett Johansson, Mark Ruffalo and Jeremy Renner.
Evans said on Good Morning America: "I kind of give a lot of credit to [Robert] Downey [Jr]. He's kind of the father of this group. If he hadn't done so well with those Iron Man movies, I don't think this movie would have happened."
The 30-year-old continued: "He came to the set with such a positive attitude and has such a great demeanor. We all kind of fell in line as a result of his disposition. He's just an amazing guy and he made it a really family atmosphere right away."
Hemsworth admitted that the rest of the cast were satisfied with their smaller roles compared to Downey Jr's part in the motion picture.
The 28-year-old explained: "I think we all preferred it, to be honest. When you do your individual films the responsibility rests on your shoulders whereas, this, you get to share the workload. And I'm happy to do that any day. It makes it much easier."
The Avengers - which is already showing in the UK under the title Marvel Avengers Assemble - is released in the US on May 4.