A major outstanding challenge is how to observationally infer the formation of black hole mergers using gravitational waves alone. I will discuss and show how several likely formation channels will give rise to modulations of the gravitational wave signal, which can be modeled and looked for with current (LIGO/Virgo) and future (3G) ground-based observatories. Modulations of the gravitational wave signal might be the only way to distinguish the growing list of suggested formation pathways apart with gravitational waves alone. Modeling and looking for such modulations, i.e. deviations from 2-body isolated vacuum inspirals, will start to play a key-role in the coming years when sensitivity increases.