Let $X$ be a compact K\"ahler manifold and $(E,\overline\partial_E,\theta)$ be a Higgs bundle over it. We study the structure of the Kuranishi space for the pair $(X, E,\theta)$ when the Higgs bundle admits a harmonic metric or equivalently when the Higgs bundle is polystable and the Chern classes are 0. Under such assumptions, we show that the Kuranishi space of the pair $(X,E,\theta)$ is isomorphic to the direct product of the Kuranishi space of $(E,\theta)$ and the Kuranishi space of $X$. Moreover, when $X$ is a Riemann surface and $(E,\overline\partial_E,\theta)$ is stable and the degree is 0, we show that the deformation of the pair $(X,E,\theta)$ is unobstructed and calculate the dimension of the Kuranishi space.