Ligand-protected gold clusters with an asymmetric nature have emerged as a novel class of chiral compounds, but the origins of their chiroptical activities associated with helical charge movements in electronic transitions remain unexplored. Herein, we perform experimental and theoretical studies on the structures and chiroptical properties of Au-13 clusters protected by mono- and di-phosphine ligands. Based on the experimental reevaluation of diphosphine-ligated Au-13 clusters, we show that these surface ligands slightly twist the Au-13 cores from a true icosahedron to generate intrinsic chirality in the gold frameworks. Theoretical investigation of a monophosphine-ligated cluster model reproduced the experimentally observed circular dichroism (CD) spectrum, indicating that such a torsional twist of the Au-13 core, rather than the surrounding chiral environment by helically arranged diphosphine ligands, contributes to the appearance of the chiroptical response. We also show that the calculated CD signals are dependent on the degree of asymmetry (torsion angle between the two equatorial Au-5 pentagons), and provide a visual understanding of the origin of helical charge movements with transition-moment and transition-density analyses. This work provides novel insights into the chiroptical activities of ligand-protected metal clusters with intrinsically chiral cores.