Final month, we began sharing the DevOps journey at Microsoft by means of the tales of a number of groups at Microsoft and the way they strategy DevOps adoption. As the following story on this sequence, we wish to share the transition one group comprised of a basic operations position to a Web site Reliability Engineering (SRE) position: the story of the Xbox Reliability Engineering and Operations (xREO) group.
This transition was not simple and got here out of necessity when Microsoft determined to carry Xbox video games to avid gamers wherever they’re by means of cloud sport streaming (mission xCloud). So as to ship cutting-edge expertise with top-notch buyer expertise, the group needed to redefine the way in which it labored—bettering collaboration with the event group, investing in automation, and get entangled within the early levels of the applying lifecycle. On this weblog, we’ll evaluation among the key learnings the group collected alongside the way in which. To discover the complete story of the group, see the journey of the xREO group.
Constant gameplay necessities and the necessity to collaborate
A constant expertise is essential to a profitable sport streaming session. To make sure avid gamers expertise a sport streamed from the cloud, it has to really feel like it’s operating on a close-by console. This implies making a globally distributed cloud answer that runs on many knowledge facilities, shut to finish customers. Azure’s international infrastructure makes this attainable, however working a system operating on prime of so many Azure areas is a critical problem.
The Xbox builders who’ve began architecting and constructing this expertise understood that they may not simply construct this method and “throw it over the wall” to operations. Each groups needed to come collectively and collaborate by means of your entire software lifecycle so the system may be designed from the beginning with issues on how will probably be operated in a manufacturing surroundings.
Architecting a cloud answer with operations in thoughts
In lots of giant organizations, it is not uncommon to see improvement and operation groups working in silos. Builders don’t all the time take into account operation when planning and constructing a system, whereas operations groups should not empowered to the touch code despite the fact that they deploy it and function it in manufacturing. With an SRE strategy, system reliability is baked into your entire software lifecycle and the group that operates the system in manufacturing is a valued contributor within the planning section. In a brand new strategy, involving the xREO group within the design section enabled a collaborative surroundings, making joint expertise selections and architecting a system that would function with the necessities wanted to scale.
Leveraging containers to obviously outline possession
One of many first technological choices the event and xREO groups made collectively was to implement a microservices structure using container applied sciences. This allowed the event groups to containerize .NET Core microservices they’d personal and take away the dependency from the cloud infrastructure that was operating the containers and was to be owned by the xREO group.
One other technological choice each groups made early on, was to make use of Kubernetes because the underlying container orchestration platform. This allowed the xREO group to leverage Azure Kubernetes Service (AKS), a managed Kubernetes cloud platform that simplifies the deployment of Kubernetes clusters, eradicating lots of the operational complexity the group must face operating a number of clusters throughout a number of Azure areas. These joint selections made possession clear—the builders are chargeable for all the pieces contained in the containers and the xREO group is chargeable for the AKS clusters and different Azure providers make the cloud infrastructure internet hosting these containers. Every group owns the deployment, monitoring and operation of its respective piece in manufacturing.
This sort of strategy creates clear accountability and permits for simpler incident administration in manufacturing, one thing that may be very difficult in a monolithic structure the place infrastructure and software logic have code dependencies and are onerous to untangle when issues go sideways.
Scaling by means of infrastructure automation
One other finest apply the xREO group invested in was infrastructure automation. Deploying a number of cloud providers manually on every Azure area was not scalable and would take an excessive amount of time. Utilizing a apply generally known as “infrastructure as code” (IaC) the group used Azure Useful resource Supervisor templates to create declarative definitions of cloud environments that permit deployments to a number of Azure areas with minimal effort.
With infrastructure managed as code, it can be deployed utilizing steady integration and steady supply (CI/CD) to carry additional automation to the method of deploying new Azure sources to present knowledge facilities, updating infrastructure definitions or bringing on-line new Azure areas when wanted. Each IaC and CI/CD, allowed the group to stay lean, keep away from repetitive mundane work and take away a lot of the threat of human error that comes with guide steps. As an alternative of spending time on guide work and checklists, the group can give attention to additional bettering the platform and its resilience.
Web site Reliability Engineering in motion
The journey of the xREO group began with a have to carry one of the best buyer expertise to avid gamers. It is a nice instance that reveals how groups who wish to delight clients with new experiences by means of innovative innovation should evolve the way in which they design, construct, and function software program. Shifting their strategy to operations and collaborating extra carefully with the event groups was the true transformation the xREO group has undergone.
With this new mindset in place, the group is now properly positioned to proceed constructing extra resilience and additional scale the system and by so, ship the promise of cloud sport streaming to each gamer.
