Description:
You will own the path a request takes from the API edge to the accelerator and back. That means the Go gateway that fronts our models, and the serving engines behind it.
Concretely:
- Extend and operate our Go API gateway: routing, authentication, rate limiting, quota enforcement, per-model metering, streaming, request shaping, failure handling.
- Deploy and tune open-weight models on our Ascend 910B clusters using the CANN/MindIE stack, and on vLLM and SGLang where applicable.
- Own serving performance: continuous batching, KV cache configuration, tensor and expert parallel layouts, prefill and decode behaviour, context length trade-offs.
- Run quantisation work. Our path is BF16 to INT8 (W8A8) to INT4 (W4A16). You will run the conversions, measure the quality cost, and be honest about it.
- Work within a hard architectural constraint: an 8-chip HCCS coherent domain per cluster. Large MoE models have to fit that shape or be sharded around it. This is the interesting part of the job.
- Build and operate retrieval infrastructure for enterprise deployments, including vector stores, embedding endpoints and the ingestion path around them.
- Instrument everything. Per-endpoint latency, tokens per second, time to first token, utilisation, cost per million tokens. If it is not measured we do not claim it.
- Support dedicated enterprise deployments, including air-gapped and restricted-network environments where you cannot assume internet egress or a friendly package manager.
What we need you to already have
- Strong Go. You have written and operated production services in it, not just read about it.
- Real experience serving LLMs in production with vLLM, SGLang, TensorRT-LLM, TGI or equivalent. You should be able to explain why throughput fell when concurrency rose, and what you did about it.
- A working mental model of transformer inference: attention, KV cache, batching, the prefill and decode split, why MoE routing changes the memory picture.
- Linux, containers, and comfort at the level of drivers, kernels and hardware topology. This stack breaks below the Python line.
- Python for model and evaluation work.
- The habit of measuring before claiming. We mark unverified numbers as unverified, in customer documents and internally.
What will help but is not required
- Huawei Ascend, CANN or MindIE experience. Very few people have this. If you do, say so early.
- Quantisation experience, particularly INT8 and INT4 on non-NVIDIA silicon.
- Kubernetes, and experience running stateful GPU or NPU workloads on it.
- Experience with regulated deployments, audit logging, or data residency requirements.
- Open source contributions to any inference or serving project.