The Amiga's way of getting relative pointers to libraries was amazing (and quite ahead of its time). Like you said, the only absolute address that always worked was 4, exec.library, and after that, you used that pointer to open other libraries, guaranteeing that they could be relocated any time.