I am having the same problem. Spent 4 hours trying to get PXE boot working. It seems that esx 5.1 does not really work correctly. Tested it with tftp on Oracle Enterprise Linux but also created a new pxelinux (6.0.1). Same error ultimately, although the menu appeared.
Suggestion to vmware: remove from manual when not working.