BurtSherry,
Normally, the Server Properties is where you can define custom sizes, so maybe you should try to recreate the custom page size and see if it takes it. Also close and restart PrintShop Mail to make sure that it will reload the list of sizes properly.
If that doesn't work, you can try the PostScript Custom Page Size, though as you've already figured out, PrintShop Mail doesn't treat it any differently that another page size(it's not "custom", as far as PSM is concerned). To configure it, you need to go to the Advanced settings, and set the custom size there. It should then take it.
However, if that still doesn't work, then it could be that the driver isn't passing standatd PostScript options, and works differently. (one way you could confirm this is by trying a different driver, and see if it sees the custom size...no need to actually try printing, just see if it takes the options, to confirm that it's a driver issue) Perhaps the Fiery configuration options are not recognised by PrintShop Mail as printer postscript commands, preventing it from recognising them and using them. With that said, I would still try to actually print, and see how it comes out, as it might still end up using the correct size if you configured it in the driver. But to see it in PrintShop Mail, you might need to use a pure PostScript driver, if available. (you might want to check your manufacturer's web site, and see if there's alternative drivers that you can use)
Let us know how it goes!
Regards,
Raphaël Lalonde Lefebvre