Delay with Time-to-byte apache configuration problem

I just cant the issue of random 7-10 seconds time-to-first-byte. The script is hosted on a amazon server. The random 7-10 seconds is occurring to random files at random timing.

http://goo.gl/AqE56

Possible due to some very limited connection per client? Which doesnt make sense too, 7 seconds is too much.

http://goo.gl/UfeCw

Any suggestions will be greatly appreciated. I think there are issues with configuration

Problem does not occur on hostgator servers**