The operator formalism for Chern-Simons gauge theory with gauge group SU(N) is presented. The connection with rational conformal field theory is shown explicitly by identifying a basis for the Hilbert space of the theory with the set of characters corresponding to a Wess-Zumino-Witten model for SU(N). Knot operators are constructed performing the calculation of matrix elements of Wilson line operators on this Hilbert space. Using these operators a representation of the Verlinde operators in the context of Chern-Simons gauge theory is obtained. As an application of the use of these operators to knot theory, the Jones polynomial for toral knots is explicitly computed.