I just watched the first episode season 4, and that is when Castiel shows up. That's the most recent season, so it sounds like you got halfway through last season.

Yeah, GoF wasn't very good either...I don't think I bothered rewatching that, now that you mention it...