Hitting the magazine rewards 10% to 50% damage in addition to the citadel damage (so you deal 110~150% damage.)
Hitting the magazine rewards a 0.07% to 75% chance of detonation depending on damage (destroying a module like a turret is between 0.01% and 33% chance of detonation depending on the amount of HP the module had1,) but it doesn't guarantee a detonation. Hitting the magazine also can destroy or incapacitate modules of any kind (main turrets, secondary turrets, torpedo bays, AA turrets2.)
Magazine hitbox is located inside the citadel hitbox, so you cannot hit the magazine hitbox without hitting the citadel (this is where the 10~50% damage bonus comes from, it acts as a second normal penetration)
1. This is why sometimes a random HE shell can detonate you, it destroyed an AA or secondary turret which caused some damage to the magazine which ultimately ended up detonating you. Dealing damage to modules gets transferred over to the magazine hitbox which has less HP than the main ship, but once it reaches 0 HP you have a 75~90% chance of detonation depending on the size of the ship and how many magazine hitboxes there are. This is also why you can't deal damage via destroyed modules anymore, since the HP for the module cannot go lower than 0, it doesn't transfer any to the magazine hitbox and in turn doesn't damage the ship.
2. This is how you can lose some turrets on the other side of the ship when you get citadelled, like modules can damage the magazine hitbox, it can also go the other way the damage is symbolic. This is also why we saw a small rise in detonations after the turret HP buff, once destroyed it dealt more damage to the magazine hitbox at once causing a significantly higher chance of detonation
The detonation mechanic is actually fairly complex, but it's still largely dice rolling.