The "trick" to the problem is to recognize that you are trying to maximize revenue (r), not the number of books sold. You do not have an equation for revenue, but we know that the revenue is equal to the price times the quantity: r = p * q

Hence, since:

r = p * q
r = q * (sqrt(1152-4q^2))

To find the maxima, you set the first derivative to zero and solve (dr/dq = 0 and solve for q). That tells you the q that maximizes the revenue. Then substitute the maxima q back into the price formula and you have your price.

Edit: I left out a ). Let me know if you want the answer so your buddy can check his work.