"By the end we were like, dude, it was hard to hang in a room, even though we loved each other. The vibes were down bad," Seavey said