I dont care if the power supply was made by God and can supply a billion watts. If the wires connecting the supply to the load do not have a high enough current carring capacity for the total load current, one, some, or all of the devices that make up the total load will have a lower input current than the load requires for normal rated operation. This will cause the performance of one, some, or all of the load devices to suffer. Some effected loads may be unable to operate at all, and some of the effected loads may operate below the manufacturer's ratings.
The audio amp in question was able to operate closer to normal parameters when a load was disconnected from it. This suggests that the amp did not have enough input current to divide across its loads while still driving them at normal parameters. An increase in input current should result in an increase in current divided across the loads, driving them at normal parameters.
Try powering a Pi 3 with a 2.4 amp usb port power brick. Sure, the power brick can supply enough power, but if you try to use a normal usb cable your Pi will be uncurrented. Why? The normal usb cable is not rated for 2.4 amps, because its wires are too small.