To identify the common and specific structural basis of bipolar depression (BD) and unipolar depression (UD) is crucial for clinical diagnosis. In this study, a total of 85 participants, including 22 BD patients, 36 UD patients, and 27 healthy controls, were enrolled. A voxel-based morphology method was used to identify the common and specific changes of the gray matter volume (GMV) to determine the structural basis. Significant differences in GMV were found among the three groups. Compared with healthy controls, UD patients showed decreased GMV in the orbital part of the left inferior frontal gyrus, whereas BD patients showed decreased GMV in the orbital part of the left middle frontal gyrus. Compared with BD, UD patients have increased GMV in the left supramarginal gyrus and middle temporal gyrus. Our results revealed different structural changes in UD and BD patients suggesting BD and UD have different neurophysiological underpinnings. Our study contributes toward the biological determination of morphometric changes, which could help to discriminate between UD and BD.