Symmetry of atomic lattices determines a variety of fundamental optical and electronic properties such as second harmonic generation (SHG) and piezoelectricity. The diversity of layered materials presents a unique opportunity to control the lattice symmetry in low dimension. We present our recent discoveries of intrinsic out-of-plane asymmetric structures in atomically thin layered materials. The strong covalent component of the chemical bonding in these materials maintains the vertical polarization, which is usually electrostatically unstable. We further prove that the polarization is reversible by electric field in quintuple layers with SHG and piezoresponse. Our results have potential for 2D memories and switchable electrochemistry.