Understanding the physics of galaxy formation is an outstanding problem in modern astrophysics. Recent cosmological simulations have demonstrated that feedback by star formation, supernovae and active galactic nuclei appears to be critical in obtaining realistic disk galaxies and to slow down star formation to the small observed rates. However the particular physical processes underlying these feedback processes still remain elusive. In particular, these simulations neglected magnetic fields and relativistic particle populations (so-called cosmic rays). Those are known to provide a pressure support comparable to the thermal gas in our Galaxy and couple dynamically and thermally to the gas, which seriously questions their neglect. After introducing the underlying physical concepts, I will present our recent efforts to model cosmic ray physics in galaxy formation. I will demonstrate that cosmic rays play a decisive role on all scales relevant for the formation of galaxies, from individual supernova remnants to the interstellar medium up to scales relevant for entire galaxies. Finally, I will discuss the non-thermal radio and gamma-ray emission of Milky-Way like galaxies and how the next-generation instruments can be used to infer properties relevant for galaxy formation.