Probably because it didn't quite happen that way.
The Squadron Sinister
These four villains might have looked familiar to comic readers of the time...
After his Avengers appearance, Dr. Spectrum returned to face Iron Man in Iron Man #63-#66 (October, 1973 to February, 1974 by Mike Friedrich and George Tuska), wherein the origin of the Power Prism was revealed, and it tried to take over Tony Stark (thankfully, Thor was able to stop that by smashing the Power Prism).
...but instead found himself allied himself with a witch doctor, and fought the Thing, Black Panther and Brother Voodoo, ending up dead (all in Marvel Two-In-One #40-#41, June and July, 1978, by Roger Slifer and Ron Wilson).
Evangelist Billy Roberts ended up with the Power Prism next, and faced off against the Defenders instead...in Defenders #13-#14 in May and July, 1974 by Len Wein and Sal Buscema and Giant-Size Defenders #4, (April, 1975 by Steve Gerber and Don Heck) losing the Power Prism...
...but trying to get it back in Avengers Annual #8 (in 1978 by Roger Slifer and George Perez, where Yellowjacket had been putting the pieces back together as a gift for the Wasp....which ended up with her briefly being Dr. Spectrum until the Power Prism got imbedded into Thor's hammer, which disappeared after Thor dropping it...).
...then as a part of Dr. Alice Nugent with New Thunderbolts #16 (February, 2006) with the group reuniting for a time again...
...though the other members had undergone some changes as well, one in particular, changing his name and costume pretty quickly!
The Whizzer returned to battle the Defenders when their foe, Nebulon recruited him for Defenders #13-#14, and again in Giant-Size Defenders #4, and had his memory erased by Dr. Strange, getting his old job back at New Jersey's Hudson Pharmaceuticals, until his memory was restored in Avengers Annual #8.
Radioactive Man, and yet occasionally committing crimes in his old identity of the Whizzer, which got him kicked out of the Thunderbolts, and he found himself back with the Squadron Sinister and working with the Grandmaster again).
The Speed Demon also later joined a new Sinister Six (proving this speedy villain had time to run around with most of Marvel's villain groups, which made him a little bit different than the next member of the Squadron Sinister, who switched his way to a more heroic path....).
Nighthawk became a very important member of the Defenders, being their for their battles against the Badoon (with the Guardians of the Galaxy), the Headmen, Scorpio, getting his own foe in the Ringer in Defenders #51, going crazy with the over-expansion of the team in "Defenders For A Day", splitting up the team, reuniting with the team, and then dying...
This Hyperion rejoined the Squadron Sinister for Defenders #13-#14, and again in Giant-Size Defenders #4, losing his memories thanks to Dr. Strange, then regaining them in Avengers Annual #8, before being used as a pawn by the alternate world scientist Master Menace (Emil Burbank) to fight his world's Hyperion (of the Squadron Supreme, the heroic other world group that inspired the Grandmaster to create the Squadron Sinister....who'd you think was the inspiration?) in Thor #280 by Don & Maggie Thompson, Roy Thomas, Wayne Boring and Tom Palmer.
Well, isn't that just super, man?