But so is 100°F completely fine with a shirt and shorts and some shade.

So basically: |0°F| > |100°F|, where | is the mathematical absolute operator.

0°F is really cold, while 100°F is merely somewhat hot.