I've seen this said many times but I've never known the cause of it. Is it the water becoming dirty too quickly that stunts the fish? The lack of space for them to swim? What is it?

It's the hormones that the fish release: they build up over time.

There isn't any way to test for these hormones, like how you can test the water for ammonia and all that. In addition, the only way to get rid of them is by doing a water change.