Porous Anodic Alumina (PAA) membrane was functionalized with CoFe2O4 nanoparticles and used as a substrate for the growing of very long helical-structured Carbon Nanotubes (CNTs) with a diameter less than 20 nm. The structures and morphologies of the fabricated nanostructures were characterized by field emission-scanning electron microscopy (FE-SEM), energy dispersive X-ray (EDX), and Raman spectroscopy. By uploading the CNTs on PAA, the characteristic Raman peaks of CNTs and PAA showed 4 and 3 times enhancement, respectively, which leads to more sensitive Surface-Enhanced Raman Spectroscopy (SERS) substrates. For comparison, PAA and CNTs/PAA arrays were used as SERS substrates for the detection of Hg2+, Cd2+, and Pb2+. The proposed sensor demonstrated high sensitivity and selectivity between these heavy metal ions. CNTs/PAA sensor showed excellent selectivity toward Pb2+ over other metal ions, where the enhancement factor is decreased from similar to 17 for Pb2+ to similar to 12 for Hg2+ and to similar to 4 for Cd2+. Therefore, the proposed CNTs/PAA sensor can be used as a powerful tool for the determination of heavy metal ions in aqueous solutions.