Artificial Intelligence Engineer

Artificial Intelligence Engineer

This range is provided by Tribus. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range

A$180,

• / yr - A$200,
• / yr

Direct message the job poster from Tribus

Founder | Recruiter @ tribus - connecting leading technology experts with top financial services firms across APAC | part-time recruiter, full-time...

Software Engineer - AI

LLM | Python | AWS

We're partnering with a fast-growing software company building AI-driven products used in high-stakes, real-world workflows.

The focus is on production-quality AI : systems that must be reliable, measurable, and safe at scale.

They're looking for a Software Engineer with AI experience to join a team responsible for the core AI platform, with a particular emphasis on LLM evaluation, observability, and reliability.

This is a hands-on engineering role, sitting close to product and domain experts, where your work directly influences how AI quality is defined, measured, and enforced in production.

What you'll work on

Building and operating LLM evaluation pipelines that assess model quality, robustness, and safety

Defining test sets, metrics, and evaluation workflows, including human-in-the-loop processes where required

Translating product and domain constraints into concrete, testable evaluation criteria

Running and orchestrating distributed evaluation workloads on AWS, including monitoring compute usage

Analysing evaluation results, identifying failure modes, and collaborating on mitigations (prompt changes, data updates, model selection or fine-tuning)

Integrating and assessing open-source and vendor evaluation frameworks, writing glue code where needed

Contributing to the evolution of the AI evaluation and platform architecture

What they're looking for

Experience monitoring and evaluating LLM-based applications

Hands-on exposure to LLM evaluation tools, benchmarks, and metrics

Understanding of common LLM failure modes (e.g. hallucination, bias, toxicity, prompt injection)

Experience with cloud ML infrastructure, ideally AWS

Familiarity with distributed workloads (e.g. Ray, AWS Lambda, or similar)

Comfort working with an evolving LLM observability and evaluation stack

Ability to work with non-ML stakeholders and convert qualitative requirements into quantitative tests

Working environment & benefits

Flexible hybrid setup, with twice-weekly collaboration in a modern CBD office

Strong learning and career development opportunities in a scaling business

Wellness focus including additional leave and gym membership

Collaborative team culture with regular social events

Pool table, snacks, and a genuinely supportive environment

This role is well suited to engineers who care about AI reliability and correctness, and who want to work on systems where evaluation and safeguards genuinely matter.

Must be based in Sydney with full working rights. Remote working or sponsorship is not available for this role.

Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Engineering and Information Technology

Industries

Technology, Information and Media

#J-