How are you keeping track of those?

As I recall, when we hit the ELBs connection limits the failed connections were reflected in the SpilloverCount metric.

I believe we only hit the security group limits in load testing. There, we could see connections fail to an instance when that instance's established connections as reported by netstat or similar tools hit a certain threshold.