The "Cancel Remaining Lines" checkbox will cancel the lines that are in Entered or Booked status and not the Order Header.
Sample of Cancellation:
If an Order has 6 lines (2 are Ship Confirmed and 4 are Booked,) then cancelling with "Cancel Remaining Lines" will Cancel the 4 lines that are in Booked status ONLY. It will not cancel the Order Header nor the Lines that are Ship Confirm.
Order has 6 lines and all are in Entered status. If the end requirement is to Cancel the Order Header and Lines, then Cancel the Order by using the following navigation --
Order Header -> Action -> Cancel
Order Header -> Action -> Cancel Remaining Lines, as this will cancel the 5 lines in Entered status and not the header.