The eccentricity matrix epsilon(G) = (epsilon(uv)) of a graph G is constructed from the distance matrix by keeping each row and each column only the largest distances with epsilon(uv) = {d(u, v), if d(u, v) = min{epsilon(u), epsilon(v)}, 0, otherwise, where d(u, v) is the distance between two vertices u and v, and epsilon(u) = max{d(u, v) vertical bar v is an element of V (G)} is the eccentricity of the vertex u. The epsilon-eigenvalues of G are those of its eccentricity matrix. In this paper, employing the well-known Cauchy Interlacing Theorem we give the following lower bounds for the second, the third and the fourth largest E-eigenvalues by means of the diameter d of G: xi(2)(G) >= {-1, if d <= 2; alpha d, if d >= 3, xi(3)(G) >=-d, and xi(4)(G) >= -1-root 5/2 d, where alpha = 0.3111+ is the second largest root of x(3) - x(2) - 3x + 1 = 0. Moreover, we further discuss the graphs achieving the above lower bounds.