Principal Engineer, CAPE at CRUSOE — San Francisco, CA

Conductor is the platform we're building: a self-driving fleet.

Most infrastructure teams watch dashboards and replace dead nodes after they fail. We're building something different — a control plane that predicts, decides, and acts on its own, keeping tens of thousands of accelerators doing useful work at the highest goodput in the industry while optimizing power and cost in real time. The mandate for this role is to make the fleet self-driving.

This is a Principal Engineer role reporting into Cloud Availability, working directly on one of the highest-priority technical charters in the org.

Your Charter — The Problems You'll Own

• Unified observability plane. One pane that correlates GPU, networking (InfiniBand/RoCE), storage, orchestration, and workload signals — so any engineer can diagnose and recover jobs fast.
• The fleet as one logical computer. Tens of thousands of accelerators across sites behaving as a single programmable system: one health model, one scheduler, one source of truth.
• Closed-loop autonomy, not alerting. Diagnose, decide, and remediate with no human in the loop — drain, checkpoint, replace, and resume a live job automatically. The hard part: earning enough trust to act on a running job.
• Goodput as an objective function. Don't just measure useful compute — continuously maximize it, trading scheduling, placement, and maintenance decisions against it as the north-star metric.
• Predict failures hours ahead, not seconds after. Forecast GPU, NVLink, optics, and thermal degradation before it stalls a job, and pre-emptively migrate work. An ML problem on noisy hardware telemetry at fleet scale.
• Straggler & silent-failure detection. One slow GPU stalls an entire distributed job — isolate the exact rank, GPU, and node from collective-operation signals and surface root cause fast.
• Energy-aware compute. Because we own the power stack, schedule, throttle, and place workloads against real-time energy availability, cost, and thermal headroom. The problem you can work on here and nowhere else.
• A digital twin of the fleet. Simulate failures, scheduling policies, and remediation logic before they touch production — the foundation that makes real autonomy safe.
• Agentic operations. An agent that doesn't just answer "why is this job slow" but proposes and executes the fix, with guardrails and a full audit trail.
• Zero-trust, fully auditable multi-tenancy. Every action — human or autonomous — identity-scoped, policy-checked, and streamed to customer audit systems in real time.
• Self-qualifying hardware. New and repaired nodes prove themselves through automated burn-in before taking customer load. Fleet growth becomes a software-gated pipeline.

Why This Role Is Different

• Scale almost no one gets to work on — distributed systems and control theory across tens of thousands of accelerators.
• A problem unique to Crusoe — we own the stack from energy to compute, so the platform can treat power as a control input, not a constraint.
• Greenfield authority — you define the operating standard for how we run AI at scale. You don't inherit one.

What You'll Bring

• 10+ years building infrastructure-layer systems at scale — fleet management, distributed control planes, scheduler internals, or hardware lifecycle automation. This is a systems-builder role, not a consumer of managed cloud services.
• Deep experience with distributed systems design: consensus, state reconciliation, closed-loop automation, and systems that make autonomous decisions against live production infrastructure.
• Hands-on fluency with GPU/HPC infrastructure — GPU health telemetry, NVLink/InfiniBand/RoCE fabrics, thermal and power behavior at the hardware level.
• Track record of designing and shipping large-scale observability or telemetry platforms that correlate signals across compute, network, and storage layers.
• Comfort operating in ambiguity and defining the architecture and standards for a system that doesn't exist yet — this is a 0→1 charter, not a maintenance role.
• Strong software engineering fundamentals in at least one systems language (Go, Rust, C++, or similar) and the judgment to know when to build vs. adopt existing tooling.
• Experience applying ML/statistical methods to noisy operational telemetry (failure prediction, anomaly detection) is a strong plus.
• Prior exposure to zero-trust or policy-based multi-tenancy architectures is a plus but not required.

Benefits

• Competitive compensation and equity packages
• Restricted Stock Units
• Paid time off, paid holidays & leave of absence programs
• Comprehensive health, dental & vision insurance
• Employer contributions to HSA account
• Paid parental leave
• Paid life insurance, short-term and long-term disability
• Professional development & tuition reimbursement
• Mental health & wellness support
• Commuter benefits (parking & transit)
• Cell phone stipend
• 401(k) Retirement plan with company match up to 4% of salary
• Volunteer time off
• Global travel insurance & emergency assistance
• Daily meals allowance
• Additional perks & programs specific to location

Compensation Range

Compensation will be paid in the range of up to $285,000 - $335,000 + Bonus. Restricted Stock Units are included in all offers. Compensation to be determined by the applicant's knowledge, education, and abilities, as well as internal equity and alignment with market data.