It originates in the constraints of hardware, particularly 8 and 16-bit computers with typically 4K or 64K addressable memory. Starting with the modification of existing games, it became a challenge to push how impressive a program could be made (typically an audiovisual demonstration "demo"). Such hardware becoming obsolete added to the impressiveness.
Demo and game competitions without constraints exist, but naturally become a competition of art and design, where use of existing engines is rewarded, discouraging programmers. The constrained competitions can be more appealing to programmers, where programming creativity is better rewarded.
Just a bit of historical update - most 8-bit machines topped out at 64k of memory (8 bit data, 16 bit address space), but some could do more with extra hardware or "tricks" (paging a special 8k block using a custom MMU or similar); even then, memory tended to be limited just due to cost if nothing else.
16-bit machines usually had a larger address space; the Motorola 68000 had a 24-bit address bus, for instance. While that allowed for quite a bit of memory, again (at least for most consumer hardware) most people were limited to at first 512k to 1Mb, then as the early-90s wore on, up to about 8Mb - at which point 32-bit cpus became affordable.
So the only place you really saw larger amounts of memory (and this is more my understanding than experience, as I was just a kid then) with 16-bit machines was usually businesses and other "larger scale" computing that had the need and finances to afford it. As always, I suppose.
Demo and game competitions without constraints exist, but naturally become a competition of art and design, where use of existing engines is rewarded, discouraging programmers. The constrained competitions can be more appealing to programmers, where programming creativity is better rewarded.