The Brigadier is a companion that is seen all through classic who. He is all along the Doctor’s timeline. The fourth Doctor no longer was no longer bound to earth like Pertwee’s doctor. So he left the Brigadier, well at least he wasn’t in every episode anymore. The audience was deprived of the Brigadier but the Brigdier deprived fans from Tom Baker dressing up as a viking as the Doctor. So it evens out, right?