Prime Minister Anthony Albanese pleaded with Australians not to panic over higher fuel prices, in a rare national address on Wednesday.