Designs, implements, and optimizes components in distributed systems with an emphasis on scalability, resiliency, and operability. Delivers features and load/performance tests; leverages data plane platforms and distributed state tools for high-volume retrieval, storage, and processing; and reviews peers’ implementations for scalability compliance. Builds fault-tolerant paths (redundancy, replication, automatic failover), applies recovery‑oriented principles, and implements retries, circuit breakers, and timeouts. Proactively detects and mitigates issues via tests, alarms, dashboards, and telemetry; authors runbooks and participates in incident response and RCAs. Implements standard replication and synchronization, develops automation/IaC for troubleshooting and maintenance, and applies advanced security controls (encryption, access, remediation) while ensuring change, compliance, and documentation standards are met.
Internal Responsibilities
Key Responsibilities:
System Design & Validation Development
- Design and develop scalable validation frameworks for GPU racks and HPC infrastructure.
- Build automated benchmarks and diagnostics for NCCL performance, GPU health, NVLink connectivity, and InfiniBand and high-speed networks.
- Develop Slurm-based workflows for distributed validation and performance testing.
- Implement retries, timeouts, and recovery mechanisms to improve framework resilience.
- Develop functional, load, and fault-injection tests to assess reliability and production readiness.
Reliability & Operational Engineering
- Build dashboards, telemetry, and alerts to monitor infrastructure health and validation results.
- Develop automated tools for Linux diagnostics, log analysis, hardware fault isolation, FDT, and cable validation.
- Convert incident findings into reusable diagnostics and preventive automation.
- Develop recovery runbooks and contribute to incident response and root cause investigations.
Automation, Security & Change Management
- Develop and maintain Bash, Python, and Infrastructure as Code tools for deployment and configuration.
- Implement access controls and secure handling of infrastructure credentials and diagnostic data.
- Automate patching, updates, and rollback workflows in accordance with change management practices.
- Maintain technical documentation and improve automation coverage across infrastructure workflows.
Core Responsibilities:
Planning & Delivery
- Translate project requirements into development plans and deliverables.
- Manage assigned work with minimal supervision, meeting agreed timelines and quality standards.
- Adapt priorities as requirements, resources, and dependencies change.
Collaboration & Problem Solving
- Work with HPC, infrastructure, and engineering teams to define requirements and align on outcomes.
- Use logs, metrics, and diagnostic results to investigate issues and develop lasting solutions.
- Resolve issues independently, escalate when needed, and document findings to support shared understanding.
Continuous Learning & Improvement
- Build expertise in GPU systems, HPC technologies, and software development through training and feedback.
- Recommend and implement improvements to validation coverage, reliability, and development efficiency.
- Share reusable tools and best practices, and incorporate team feedback into ongoing improvements.
External Responsibilities
Key Responsibilities:
System Design & Validation Development
- Design and develop scalable validation frameworks for GPU racks and HPC infrastructure.
- Build automated benchmarks and diagnostics for NCCL performance, GPU health, NVLink connectivity, and InfiniBand and high-speed networks.
- Develop Slurm-based workflows for distributed validation and performance testing.
- Implement retries, timeouts, and recovery mechanisms to improve framework resilience.
- Develop functional, load, and fault-injection tests to assess reliability and production readiness.
Reliability & Operational Engineering
- Build dashboards, telemetry, and alerts to monitor infrastructure health and validation results.
- Develop automated tools for Linux diagnostics, log analysis, hardware fault isolation, FDT, and cable validation.
- Convert incident findings into reusable diagnostics and preventive automation.
- Develop recovery runbooks and contribute to incident response and root cause investigations.
Automation, Security & Change Management
- Develop and maintain Bash, Python, and Infrastructure as Code tools for deployment and configuration.
- Implement access controls and secure handling of infrastructure credentials and diagnostic data.
- Automate patching, updates, and rollback workflows in accordance with change management practices.
- Maintain technical documentation and improve automation coverage across infrastructure workflows.
Core Responsibilities:
Planning & Delivery
- Translate project requirements into development plans and deliverables.
- Manage assigned work with minimal supervision, meeting agreed timelines and quality standards.
- Adapt priorities as requirements, resources, and dependencies change.
Collaboration & Problem Solving
- Work with HPC, infrastructure, and engineering teams to define requirements and align on outcomes.
- Use logs, metrics, and diagnostic results to investigate issues and develop lasting solutions.
- Resolve issues independently, escalate when needed, and document findings to support shared understanding.
Continuous Learning & Improvement
- Build expertise in GPU systems, HPC technologies, and software development through training and feedback.
- Recommend and implement improvements to validation coverage, reliability, and development efficiency.
- Share reusable tools and best practices, and incorporate team feedback into ongoing improvements.
