AI engineering for production systems
ComplxAI is an AI engineering company in Australia. The hard part of an AI project sits between a promising demo and a system the business can rely on. We build the part in between: retrieval pipelines, model routing, evaluation harnesses, observability and secure AWS deployment.
- You own the IP from day one of any paid stage
- Small senior team that does the work itself
- Every stage priced in writing before it begins
For teams who have already seen the demo
You have a proof of concept, a vendor pitch or a half-built feature. The demo looked good. Now the questions are harder, and they are engineering questions.
We take them on, greenfield or rescue. Still deciding whether AI belongs on the roadmap? Start with AI consulting. Need a whole application built around a model? See custom AI development. This page is the layer underneath both.
A good fit if…
- You lead engineering or product and have an AI system to ship or rescue
- A PoC works on curated examples but not on real traffic
- Output quality is judged by feel, not by an evaluation suite
- Token spend is rising faster than usage
- Security or data-residency questions are blocking sign-off
Four problems technical buyers bring us
Each looks like a model problem. Each is solved by engineering around the model.
The PoC that never ships
A notebook proved the idea; it cannot survive real data, real volume or a real integration. We rebuild it as a service behind queues, with infrastructure-as-code and CI/CD. The prompt is the smallest part. See why proofs of concept stall before production.
Outputs you cannot trust
Quality drifts with every prompt change and nobody can say whether the last one helped. We put a golden dataset and a regression suite in front of every release, so it ships when the numbers clear the bar.
Costs that grow faster than usage
One large model for every step is the easy choice and the most expensive one. Routing by task, caching, context trimming and per-tenant token telemetry give you a bill you can forecast.
Security questions you cannot yet answer
Reviewers ask where data goes, who can see what, and how the system resists prompt injection. We design the boundaries first, and put a human approval wherever an action has consequences.
What production AI engineering covers
Six disciplines, each with its own failure modes, all done in code, usually in your own AWS account, or on hosting we set up for you.
LLM system architecture and approval steps
What runs synchronously and what goes on a queue, where state lives, which calls may retry, and where a human approval sits as a first-class Step Functions state, with confidence thresholds deciding what waits for a person. Idempotency keys, dead-letter queues and timeouts are designed in from the start; reviewer decisions feed the evaluation set, so autonomy is earned on evidence, as in our custom AI agents.
Retrieval, vector search and embeddings
Ingestion, chunking and metadata from S3, document stores and internal systems; embedding model and chunk sizes tested on your real queries; hybrid search in Postgres with pgvector or a managed vector database; reranking, permission-aware filtering and freshness jobs; a re-embedding plan for when the model changes. Answers cite their passages.
Model routing, latency and cost
A router sends each task to the cheapest model that passes its eval, with fallbacks across OpenAI, Anthropic, Google, Bedrock and open-weight models; small models for classification and extraction, the large one for reasoning. Streaming, response and embedding caches, parallel tool calls, prompt trimming, batching for background jobs, and per-tenant token budgets with cost telemetry per feature.
Prompt, model and release management
Prompts are versioned artefacts with an owner, a changelog and an eval run attached. A model upgrade gets a regression run; rollback is one deploy away. Where fine-tuning is justified, we build the pipeline with our machine learning consulting practice.
Security, data boundaries and access control
Prompt-injection defences at the retrieval and tool boundary, PII redaction before data leaves your account, secrets management, least-privilege IAM, tenant isolation in the data layer, and an audit log of every model and tool call.
Integration architecture
APIs, webhooks and queues connecting the AI layer to Xero, HubSpot, Salesforce, document stores and internal databases, with idempotent writes, retries with backoff and reconciliation jobs. See our AI integration services.
A reference production LLM architecture on AWS
Every system differs; the shape rarely does. Not every project needs every box, but each layer needs a deliberate answer.
- Interfaceswhere requests enterWeb and mobile appsInternal toolsAPI GatewayInbound webhooksEmail and chat intakeScheduled jobs
- Orchestrationcontrol flow and stateStep FunctionsLambdaECS/Fargate workersEventBridgeSQS queuesApproval stepsIdempotency keysDead-letter handling
- Model layerinferenceModel routerBedrockOpenAI, Anthropic and Google APIsOpen-weight modelsVersioned prompt registryStructured outputsTool callingStreamingResponse cache
- KnowledgeretrievalIngestion pipelineChunking and metadataEmbeddingsPostgres + pgvectorHybrid searchRerankingFreshness jobsPermission filters
- Qualityevaluation and observabilityGolden datasetOffline eval setsRegression suite in CILLM-as-judge, human-calibratedEnd-to-end tracingToken and cost telemetryCloudWatch alarms
- Foundationsdata, security, operationsS3DynamoDBRDS/PostgresCognito and IAMSecrets managementPII handlingTenant isolationPrompt-injection defencesTerraform/CDKCI/CD
We favour serverless, event-driven designs: Lambda and Step Functions for orchestration, EventBridge and SQS so one slow model call never blocks the rest, Postgres with pgvector unless scale calls for a managed vector database, all described in Terraform or CDK. Our AWS consulting covers the platform without the AI.
How we measure quality
Before we change a prompt, swap a model or tune retrieval, we need a number that says whether it got better. So the first artefact is a golden dataset: real inputs, expected outputs and rubrics written with your domain experts.
From there: offline evaluation sets, a regression suite in CI that fails the build when quality drops, and LLM-as-judge scoring calibrated against human graders and re-checked when the judge changes. Retrieval gets its own metrics, because a bad answer from good context and a good answer from the wrong document are different bugs.
What the quality dashboard shows
- One trace per request across retrieval, models, tools and writes
- Tokens and cost per feature, tenant and model
- Latency by stage: retrieval, generation, tool calls
- Evaluation score trend across prompt and model versions
- Retrieval hit rate, citation coverage and refusal rate per release
- Human override and escalation rate
Demo-grade versus production-grade
The same feature can be built two ways. The first is faster to show; the second is the one a business can run.
| Concern | Demo-grade | Production-grade |
|---|---|---|
| Retrieval | Documents pasted into the prompt, or one vector index with no metadata. | Ingestion pipeline with chunking, metadata and permission filters; hybrid search; freshness jobs; retrieval measured separately from generation. |
| Evaluation | Someone reads a few outputs and says it looks right. | Golden dataset owned by domain experts, regression suite in CI, LLM-as-judge calibrated against human graders, release gates. |
| Observability | Console logs. Cost discovered on the invoice. | End-to-end traces per request, token and cost telemetry by feature, quality dashboards and alarms. |
| Security | API key in the code. Every user can reach every document. | Secrets management, least-privilege IAM, permission-aware retrieval, tenant isolation, prompt-injection defences, PII handling. |
| Cost control | One large model for every step. | Model routing by task, caching, structured outputs, smaller models where they pass the eval, budgets and alerts. |
| Failure handling | Retry until it works, or wait for a human to notice. | Timeouts, idempotent handlers behind queues, dead-letter handling, fallbacks and approval steps designed in. |
| Integration | Copy the answer into the system of record by hand. | Event-driven integration with your systems of record, reconciliation jobs, an audit trail of every action. |
| Ownership | Lives in a vendor's platform or on one engineer's laptop. | Infrastructure as code, usually in your own AWS account, documented and versioned, yours from day one of any paid stage. |
How an engagement runs, block by block
Every engagement follows the five-stage model in how we work: two free conversations, Discovery, Strategy, then Implementation in 12-week blocks. Inside a block, the engineering runs in this order.
- Architecture reviewCode, data, prompts and infrastructure as they stand, and what is failing.
- Reference designExecution model, data boundaries, model strategy, security model and integration contracts.
- Eval baselineGolden dataset and harness before any tuning, so every change has a score.
- BuildRetrieval, orchestration, integrations and interfaces shipped in vertical slices.
- HardenLoad, failure and security testing; cost and latency tuning; alarms wired in.
- Operate or hand overRunbooks, dashboards and documentation. We run it with you, or you take it in-house.
An AI engineering company, not a prompt wrapper
We build what we recommend
Strategy, engineering and deployment sit in one small senior team, remote-first across Australia. The architecture we propose is the one we ship.
Your IP from day one
Code, prompts, eval sets and infrastructure-as-code are yours from the first paid stage, usually in your own repositories and AWS account, or on hosting we set up for you. No mandatory retainers.
Priced per stage
Every stage is priced in writing before it begins and you can exit at any boundary. Implementation runs in 12-week blocks with working software every couple of weeks.
Systems we have shipped
Built for DisabilityAssessments: 90% of operations automated, as reported by the client. Modedu reported 2× revenue within six months. Luxpip reported manual workload down 75%. See what we've built.
Where this work connects
AI development
Custom AI applications and LLM systems.
Custom AI developmentAI agent development
Agents that act in your systems, with approvals and an audit trail.
AI agent developmentAWS consulting
Serverless, event-driven and container platforms on AWS.
AWS architectureAI integration
AI outputs connected to Xero, HubSpot, Salesforce and your databases.
Integrating AI with existing systemsMachine learning
Predictive models, MLOps and when a custom model beats an API.
Machine learning consultingAI consulting
Use-case selection and a scored roadmap.
AI consulting in AustraliaQuestions engineering leaders ask
Bring the harder ones to the intro call. It is free.
Book a free intro callDo you train models?
Not foundation models. In our experience, retrieval, routing and evaluation around an existing model get a system further than training would, so that is where we start. Where a narrow task justifies it, we fine-tune a smaller model or build a classical predictive model.
Which model providers do you use?
OpenAI, Anthropic, Google, AWS Bedrock and open-weight models, selected per task. Providers are components we engineer around, not the product: the routing layer lets you swap or mix them without a rewrite.
Can our data stay in Australia?
Yes. We deploy in the region you need; for Australian data residency that is the AWS Sydney region, where storage, databases, vector search and orchestration all run in your own AWS account, or one we set up for you. Where a model API is only served offshore we say so upfront and design redaction and data boundaries around it.
What does an engagement like this cost?
Every stage is priced in writing before it begins. The first two conversations are free; Discovery starts from a fixed fee, Strategy is priced upfront, and Implementation is priced per 12-week block with an exit at every boundary. For a rough range, try the AI project cost calculator.
Bring us the demo, the PoC or the blank page
Walk us through the architecture, the evaluation results and the bill. The first two conversations are free, and every stage after that is priced in writing.