Some go early. They're not suitably skilled to get food of they quickly figure out this isn't for them (one guy quit once before the first night.) In later seasons this group is very small.
Around the 30-50 day mark the emotional part comes into play. Quite a few quit for emotional reasons.
Once the hard-cores are left, then it comes down to food. At that point it becomes a race to see who can out-starve the others, without getting pulled for medical reasons.
