More Unions Better in Late Game
In the early game, when most monsters have single-unit attacks, it is often better to have a smaller number of unions with more units per union to increase the durability of each union. This is especially the case when you have no unit with resurrection.
On the flip side, in the late game when most enemy unions will use union-wide attacks, an attack that does 75% damage does that much damage whether you have 3 units or 5 units in a union. By this point, more unions with fewer units is usually better to help mitigate unit loss when one union goes down due to a powerful attack.
Also, having 5 unions greatly increases your flanking ability and the possibility of a rear assault if you have a remnant out. In the late game, the morale increase from the flanking gets more important. Especially when fighting war dragons, having high morale prevents them from using area-wide breath attacks. And while they will use Desperate Wail instead, your two flanks will help tip the morale back during the turn.
And finally, the more unions you have, the more union leaders you have that can unleash remnant or unique arts. Since these arts are usually tide-turning, it makes sense to maximize the likelihood they trigger. Thus, in general, when you can have 18 party members, you should be using 5 unions.