Montana Democrats thought they found a novel way to win control of the U.S. Senate—until the party faithful started fighting back.