We present ~800 days of photometric monitoring of Boyajian's Star (KIC8462852) from the All-Sky Automated Survey for Supernovae (ASAS-SN) and ~4000d of monitoring from the All Sky Automated Survey (ASAS). We show that from 2015 to the present the brightness of Boyajian's Star has steadily decreased at a rate of 6.3+/-1.4mmag/yr, such that the star is now 1.5% fainter than it was in 2015 February. Moreover, the longer time baseline afforded by ASAS suggests that Boyajian's Star has also undergone two brightening episodes in the past 11 years, rather than only exhibiting a monotonic decline. We analyze a sample of ~1000 comparison stars of similar brightness located in the same ASAS-SN field and demonstrate that the recent fading is significant at >=99.4% confidence. The 2015-2017 dimming rate is consistent with that measured with Kepler data for the time period from 2009 to 2013. This long-term variability is difficult to explain with any of the physical models for the star's behavior proposed to date.