We study TmB4, a frustrated magnet on the Archimedean Shastry-Sutherland lattice, through magnetization and transport experiments. The lack of anisotropy in resistivity shows that TmB4 is an electronically three-dimensional system. The magnetoresistance (MR) is hysteretic at low temperature even though a corresponding hysteresis in magnetization is absent. The Hall resistivity shows unconventional anomalous Hall effect (AHE) and is linear above saturation despite a large MR. We propose that complex structures at magnetic domain walls may be responsible for the hysteretic MR and may also lead to the AHE.