Computer vision-based measurements offer superior capabilities over traditional sensing systems, including high spatial resolutions, no mass-loading effects, and low cost. Recently developed phase-based motion estimation techniques can be used to measure the accurate motion signals from the local phase without speckle patterns on a structure. However, various conditions, such as complex illumination and noisy background, can generate an unstable phase, which introduces significant errors during the measurement process. In this study, an accurate, robust, and non-marker full-field displacement measurement technique was developed. The optimally designed steerable filter provided the stable and noiseless local phase, which resulted in considerable correlation with the motion in phase-based optical flow technique. Several experimental tests were performed to validate the proposed technique on a five-story structure using accelerometers and a camera. The experimental results showed that the proposed technique could be used to measure the displacement and identify the modal frequencies and full-field operational deflection shape with high accuracy.