Site Reliability Engineer Job at Xai, Memphis, TN

L0xGRXBjbG1KNS9lRlFHRjc2d2JZeEVNaFE9PQ==
  • Xai
  • Memphis, TN

Job Description

About xAI


xAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important. All engineers are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.

About the Role


As a Data Center Site Reliability Engineer (SRE) at xAI, you will play a pivotal role in ensuring the reliability, scalability, and performance of our state-of-the-art data center infrastructure, including the Colossus supercluster in Memphis—the world's largest AI training cluster with over 100,000 liquid-cooled Nvidia GPUs and plans for expansion to 1 million. This infrastructure powers advanced AI workloads, massive-scale model training, and products like Grok, enabling breakthroughs in understanding the universe. You will collaborate with cross-functional teams to automate operations, enhance observability, and maintain high availability for large-scale distributed systems. This is a hands-on technical position in a dynamic environment, offering the opportunity to tackle complex challenges at the intersection of AI, data center operations, and software reliability.

Responsibilities



  • Maintain and improve the reliability and uptime of xAI’s on-premises and cloud-based data center environments, including high-density GPU clusters for AI training.

  • Design, implement, and manage monitoring, logging, and alerting systems (e.g., Prometheus, Grafana, PagerDuty).

  • Develop and maintain infrastructure-as-code (Pulumi, Terraform) and continuous deployment pipelines (Buildkite, ArgoCD).

  • Participate in on-call rotations, respond to incidents, perform root cause analysis, and drive post-mortem processes.

  • Analyze system performance, forecast capacity needs, and optimize resource utilization for massive AI/ML workloads.

  • Collaborate with hardware, networking, and software engineering teams to design and implement resilient, scalable solutions, such as RDMA fabrics and liquid-cooling systems.

  • Create and maintain documentation and standard operating procedures.

  • Contribute to the efficiency of AI training pipelines by identifying and mitigating bottlenecks in compute, storage, and networking at unprecedented scales.

Basic Qualifications



  • Bachelor’s degree in Computer Science, Engineering, or a related technical field (or equivalent experience).

  • 5+ years in site reliability engineering, data center operations, or large-scale infrastructure management.

  • Expert-level knowledge of Kubernetes (on-prem and cloud), infrastructure-as-code tools (Pulumi, Terraform), and CI/CD systems (Buildkite, ArgoCD).

  • Proficiency in at least one systems programming language (Rust, C++, Go) and strong scripting/automation skills.

  • Deep understanding of monitoring and observability technologies.

  • Strong troubleshooting skills across hardware, networking, and distributed software systems.

  • Proven experience with incident response, including on-call rotations, rapid incident resolution, root cause analysis, and implementation of preventative measures.

  • Excellent communication and documentation skills, with the ability to share knowledge concisely and accurately.

Preferred Skills and Experience



  • Experience supporting AI/ML workloads or high-density compute environments, including large-scale GPU clusters and HPC systems.

  • Familiarity with data center electrical, cooling, and network systems, such as liquid-cooling and high-bandwidth interconnects.

  • Certifications in SRE, Kubernetes, or data center operations.

  • Experience with both on-premises and cloud infrastructure at scale.

xAI is an equal opportunity employer.

Job Tags

Full time,

Similar Jobs

Manpower

Weekend Safety Coordinator Job at Manpower

 ...Our client, a leader in the manufacturingindustry, is seeking a Safety Specialist to join their team. As a Safety Specialist, you will be part of the safety teamsupporting various infrastructural projects. The ideal candidate will have strong leadership skills, excellent... 

Walmart Inc.

(USA) Photographer Job at Walmart Inc.

 ...be both challenging and rewarding, then continue reading to see if this position grabs your interests. ECommerce Content Studio Photographer role will be a key contributor to growing our US ECommerce business. What you'll do... Youll sweep us off our feet if:... 

BWX Technologies

Experienced CNC Machinist Job at BWX Technologies

 ...continents. Our joint ventures provide environmental restoration and operations management at a dozen U.S. Department of Energy and NASA facilities. BWXT's technology is driving advances in medical radioisotope production in North America and microreactors for various... 

Andy Frain Services

Security Officer Job at Andy Frain Services

 ...Ionia Main Provide security detail as outlined in the post orders and establish working relationships with customers, local law enforcement and fire departments. Security personnel will always perform job duties with a constant awareness of surroundings, making note... 

Johnson Controls

Field Quality Engineer Job at Johnson Controls

As a Field Quality Engineer at Johnson Controls, you will be instrumental in enhancing the quality and reliability of our smart building technologies. Your role involves on-site assessments and evaluations of manufacturing processes to ensure our products meet rigorous...