Natural aggregate is commonly used in mortar and concrete. In some places in the world, the natural aggregate may no longer reach the construction demands in terms of quality or/and quantity. Hence, alternative resources should be searched. On the other hand, the disposal of steel slag (SS) resulting from the refinement of pig iron by electric arc furnace (EAF) or basic oxygen furnace (BOF) steelmaking caused a lot of serious environmental problems. In order to avoid the negative effect of this by-product on the environment and landfills, lots of studies investigated different methods to recycle it. Recycling SS as an aggregate in concrete and mortar is the optimum solution for the natural aggregate shortage. This became more attention than before and became a hot topic. However, some of these studies conflicted with the others. This review summarized the available studies focused on the effect of using SS aggregate as a partial or full replacement of natural aggregate on the properties of mortar and concrete. The negative or positive effects of SS aggregate on the workability, density, mechanical strength water absorption, permeability, porosity, chloride permeability, sulfate resistance, wetting/drying cycles, weathering/aging, freezing/thawing resistance, carbonation resistance, corrosion resistance, abrasion resistance, elevated temperature/thermal cycles resistance, shrinkage and expansion of concrete and mortar were summarized and analyzed. Furthermore, the common methods to modify the properties of SS aggregate matrices were summarized. Some statistics in addition to empirical equations obtained from regression analyses of the collected results were established.