We perform a z-band survey for an optical counterpart of the binary neutron star coalescence GW170817 with Subaru/Hyper Suprime-Cam. Our untargeted transient search covers 23.6deg^2^ corresponding to the 56.6% credible region of GW170817 and reaches the 50% completeness magnitude of 20.6mag on average. As a result, we find 60 candidate extragalactic transients, including J-GEM17btc (also known as SSS17a/DLT17ck). While J-GEM17btc is associated with NGC 4993, which is firmly located inside the 3D skymap of GW170817, the other 59 candidates do not have distance information in the GLADE v2 catalog or NASA/IPAC Extragalactic Database. Among 59 of the candidates, 58 are located at the center of extended objects in the Pan-STARRS1 catalog, while one candidate has an offset. We present location, z-band apparent magnitude, and time variability of the candidates and evaluate the probabilities that they are located within the 3D skymap of GW170817. The probability for J-GEM17btc is 64%, which is much higher than for the other 59 candidates (9.3x10^-3^-2.1x10^-1^%). Furthermore, the possibility that at least one of the other 59 candidates is located within the 3D skymap is only 3.2%. Therefore, we conclude that J-GEM17btc is the most likely and distinguished candidate to be the optical counterpart of GW170817.