With an unprecedented amount of mail suddenly coming through, the Australia Post digital applications team was relieved they had just done a peak load readiness check.
With an unprecedented amount of mail suddenly coming through during the COVID-19 pandemic lockdown due to panic buying, the Australia Post platform engineering team was relieved they had done a peak load readiness check before parcel volume hit its peak.
In April 2020, Australia Post recorded 340 million transactions—that’s double the volume in April 2019 (140 million) and 70 per cent higher than December 2019’s 200 million transactions, which is usually the peak time of the year due to Christmas shopping.
In April 2020, user registrations for tracking services also increased by 30 per cent compared to December 2019, while recorded parcel tracking numbers were 40 per cent higher than the Christmas month.
Australia Post also introduced a pharmaceuticals delivery scheme to send vulnerable customers their prescription medication free of charge. That generated a lot of new registrations. To top it all off, April marks the time of the year for PO boxes renewal for the entire country, which Australia Post head of platform engineering Andrew Nette described as a very intensive period, with just one chance to get it right.
He told Computerworld Australia that the COVID-induced spike happened during the traditional Easter peak. Just before Easter, expecting a COVID-induced wave of packages, Nette’s team went through a very rapid process of checking the systems for peak load readiness, a process that usually takes place before Christmas. This check-up ensures that servers can scale but also that the right people are available.
The two top concerns for the AusPost platform engineering team
There are two limiting factors that can affect the platform engineering team: the back-end systems and the delivery network.
Whereas the digital applications are on Amazon Web Services, the back-end systems are in the data centre in a traditional server infrastructure, which cannot scale as quickly as the cloud infrastructure can.
“There was a lot of time spent making sure that those guys knew that the load was coming and working with them on the kinds of loads we were seeing and the kinds of errors we were seeing, and they were then able to tweak and tune and do things to make sure that the back ends were able to respond as well as the cloud front end,” Nette said.
The delivery network imposes a physical challenge as there are only a certain amount of parcels that can go through the sorting machines, into the trucks and in the warehouses.
AusPost builds caching to help free up back-end systems
Nette’s team uses New Relic’s platform monitoring tools to keep all its different applications in check and visualise where issues are occurring. That allowed the team to identify an area where it had previously not needed support in the new registration flows on the small business portal.
AusPost also built caching for its transactional systems which took 80 percent of the load experienced with requests now being served by cache rather than going to the back end. The platform engineering team helped with tests to make sure all was working as needed.
Working in a devops environment allowed the operations engineers, who monitor and observe the network, to liaise with the delivery engineers who were redeployed to make sure that the environment was up and running and stable, rather than adding new features at the time.
Adapting to the pandemic’s work-from-home environment was an easy task as the team was never 100 per cent present in the office at one time but instead had a rotating work-from-home policy. The only area they had to focus on was on communication and collaboration, making sure that all the messages and troubleshooting information was in one central place that everyone could call on.
AusPost is now preparing for a busier-than-ever Christmas—and future
To improve other areas, Australia Post has been looking at rolling out the caching more broadly to help with the resilience in the digital applications network. “Our back-end teams have already started with their preparation to make sure that there’s capacity for our peak period,” Nette said.
He also noted that prior to COVID-19 they could usually anticipate the extra volume coming during Christmas, but that has now changed as the record volumes haven’t dropped yet.
Nette believes that volumes will likely never return to what they were even though they are likely to drop at some point. He believes the COVID-19 pandemic is likely going to change people’s behaviour as they get more comfortable shopping online.
