Exactly, this isn't a gotcha. This is one of the basic checks on Congress.
Congress can also impeach justices, which is one of their checks on the Judicial branch. They also have the authority to confirm the Executive's judicial nominees (one of their checks on the Executive).
But even then, with enough agreement from the states, Congress can amend the Constitution.