Well, it's quite simple. Starboard comes from the side of a boat that would be the place of steering it.
And port is quite simply a reference to the docking of a vessel as in the direction of the port. However, it was once known as larboard meaning loading side but sounded too similar to starboard so it was called port instead.Early Germanic peoples' boats were propelled and steered by a paddle on the right side.
Personally, left and right would have been suffice but being a boat means everything must be called something different.