I too seem to have the problem of having to tough my brakes alot even when i coast way before the rider before me coasts, must be a byproduct of my low drag
Can I, politely, suggest it is a function of being uncertain and not looking far enough ahead? If I get nervous - a rare event but often associated with being around a couple of people who I know to be dangerous riders, I find I tend to bring my focus in closer and then start to react to things one or two wheels ahead.
However, I agree you would have relatively low drag, but you also have relatively low inertia compared with someone like me - high drag but high inertia. If I weighed twice what you weigh - and I probably exceed that - then if all things other than your aerodynamic profile were equal then you would need to have twice the aerodynamic profile that I have, and as fat as I am and as skinny you are I doubt that would be the case.
You will notice when you ride with me, on track or road, that I will often lay off quite early simply because I know I will maintain my momentum for longer than pretty much everyone else except for Gav Vickers. On the track I tend to enter a corner with a bit of a gap because I will carry speed through the bank slightly better than a lighter rider.