Weyl semimetals are expected to open up new horizons in physics and materials science because they provide the first realization of Weyl fermions and exhibit protected Fermi arc surface states. However, they had been found to be extremely rare in nature. Recently, a family of compounds, consisting of tantalum arsenide, tantalum phosphide (TaP), niobium arsenide, and niobium phosphide, was predicted as a Weyl semimetal candidates. We experimentally realize a Weyl semimetal state in TaP. Using photoemission spectroscopy, we directly observe the Weyl fermion cones and nodes in the bulk, and the Fermi arcs on the surface. Moreover, we find that the surface states show an unexpectedly rich structure, including both topological Fermi arcs and several topologically trivial closed contours in the vicinity of the Weyl points, which provides a promising platform to study the interplay between topological and trivial surface states on a Weyl semimetal's surface. We directly demonstrate the bulk-boundary correspondence and establish the topologically nontrivial nature of the Weyl semimetal state in TaP, by resolving the net number of chiral edge modes on a closed path that encloses the Weyl node. This also provides, for the first time, an experimentally practical approach to demonstrating a bulk Weyl fermion from a surface state dispersion measured in photoemission.