Yes you can setup custom nameservers, the IP addresses will be the IP addresses of the nameservers provided in your Account Information e-mail.
You can find out the IP address of a server by opening a command prompt on your pc and typing:
ping <servername>
So if your nameservers were ns1.byethost.org and ns2.byethost.org
ping ns1.byethost.org
PING ns1.byethost.org (31.22.4.21) 56(84) bytes of data.
This shows that ns1.byethost.org has an IP address of 31.22.4.21
If you have setup custom nameservers in your domain registrar control panel please raise a support ticket with us to ensure your DNS configuration is all working correctly.
If you require any assistance on configuring your own custom nameserves please raise a support ticket.