We present a refined catalog for the positions of 179 gamma-ray burst (GRB) X-ray afterglows observed by the X-Ray Telescope (XRT) on Swift prior to 2006 November 1. The positions are determined by detecting X-ray field sources in the deep X-ray images and comparing the centroids to those of optical sources in the Digitized Sky Survey red2 catalog or the Sloan Digital Sky Survey Data Release 5 catalog. Half of the 90% confidence error region radii are <2.2". The error regions areas are typically ~4 times smaller than the best XRT team error regions, although the positions require deep X-ray integration (>20ks) and cannot be generated nearly as rapidly after the GRB. The positions derived for >90% of 77 bursts with optical afterglows are consistent with the optical transient positions, without the need for systematic error. About 20% of the afterglow positions require a sizable shift in the Swift satellite aspect. We discuss the optical/X-ray properties of the field sources and discuss the implications of the frame offsets for studies of optically dark GRBs.