It's things like bandwidth between components (EDIT and, in general, "fit" between components, and fit with the end-goal rather than with intermediate interfaces). I don't know Apple's internal design details, but they consistently out-perform competitors with higher specs. A good example is the Transformer Prime (quad-core) performing about the same as the iPad 2 (dual-core) - even though the Prime has a higher clock rate.
I think this is the review: http://www.anandtech.com/show/5163/asus-eee-pad-transformer-...
While it's true that Apple has its own SoC, this is built from existing components: it's just that the components are packaged together, instead of distributed across a mobo.
When you have a very limited set of hardware, and work side by side with the people that made it, it's far easier to optimize your software vs. one company writing software for 20 other hardware companies they don't have direct contact with.
Months ago on Hypercritical [1], John Siracusa had a series of episodes where he was talking about weaknesses of programming in Objective C vs more dynamic runtimes like C#/Java, like garbage collection, dynamic typing, etc.
One of the advantages he cited & partially dismissed, though, was that since Objective C compiles directly onto the hardware, performance & battery life is better on ram- and cpu-limited phones compared to the added overhead of Dalvik or another virtual machine or JIT. (His theory was that that bought them time in the short term but is still a long term issue for Apple; I'm not sure I agree, but don't disagree Apple needs to be evaluating this stuff.)
It probably helped more in 2007 vs. now; modern phones have a bit more horsepower to where they can afford to shrug off a VM. The narrow field of hardware helps significantly in terms of /feeling/ fast if not actually being faster. You'll note that Windows Phones are fairly snappy besides running on effectively year-old hardware and running everything* (including games, WinPhone doesn't support native code) on top of the CLR; since WinPhone only supports two SoCs (the Snapdragon 8x50 and 8x55), MS can pour all their energy into optimizing every last drop of performance out of that chipset.
* There's at least one exception in Trident (the IE rendering engine). Assuming most of the stock apps are CLR-based unless proven otherwise.
I think this is the review: http://www.anandtech.com/show/5163/asus-eee-pad-transformer-... While it's true that Apple has its own SoC, this is built from existing components: it's just that the components are packaged together, instead of distributed across a mobo.