So you are looking at a few different things here. Okay, here we go:
Pixel aspect ratio – Square pixels is just that – square pixels. This will allow you to have any pixels size you wish.
16:9 or 4:3 means it is using NON-square pixels that are either stretch side-to-side (16:9) or up and down (4:3). All HD video uses square pixels.
Aspect ratio – This allows you to override the default aspect ratio of your SOURCE file. This is only useful to change if you want to change the aspect ratio of your source file or if Squeeze is not importing it correctly. The “RAW (Square Pixels)” setting will just show the raw pixels of your source and not try to fit it into a predefined aspect ratio.
Frame size – This is the aspect ratio of your OUTPUT file. The “same as source” setting is based on the aspect ratio (see above) of your source. Unconstrained will allow you to put in any aspect ratio you want (based on the pixels you set in the frame size) Maintain aspect ratio will allow you to change the frame size but it will always alter the height to stay in the same ratio as your source. Letterbox or pillar will keep the aspect ratio of your file inside of a black box – so if your frame size is larger than your source file’s aspect ratio the rest of the frame (either above and below or the sides) will just be black pixels.
There is no way to make some sort of custom aspect ratio that you can always keep. So if you want to take a 16:9 source and make a 17:24 aspect ratio you will need to work out the frame size yourself (such as 1700 × 2400) and use Unconstrained to force your source into a box that size. You cannot set some new default “17:24” aspect ratio to use. You would need to make a new preset that has your custom Unconstrained aspect ratio saved and use that for all your videos.