AI inference and serving: run a model, test the boundary
Inference is not the whole service
Inference applies model parameters to inputs. Serving makes that operation available through an interface with validation, errors and operational limits.
Build
Run a tiny fixed-weight linear model over local HTTP, without an API key or package installation.
Measure
Compare four individual requests with one grouped request, preserving outputs and timing boundaries.
Limit
Synthetic arithmetic quality is not business accuracy. Local HTTP timings do not benchmark an LLM or cloud service.

This is a builder exercise in separating a prediction from the service that delivers it. You need basic JavaScript and a local terminal. No customer data, cloud deployment, paid model or AI account is needed. The output is a repeatable test record, not a claim that you have deployed a production AI system.
Training, inference and serving are different operations
Google’s machine-learning glossary describes inference as making predictions by applying a trained model. The input need not be new to the model. A useful prediction and a correct prediction are separate questions.
On a narrow screen, scroll each table sideways; keyboard users can focus the table and use the arrow keys.
| Operation | What changes | In this exercise |
|---|---|---|
| Training | Parameters are fitted using examples | The saved fixture has weight 2 and bias 1, fitting invented pairs (0,1), (1,3), (2,5) |
| Fine-tuning | Further training adapts an existing model | Not performed; it is not a mandatory stage before inference |
| Inference | Inputs produce predictions using the current parameters | For x = 3, the result is 2 × 3 + 1 = 7; parameters do not change |
| Serving | An interface handles requests and responses | Loopback HTTP validates JSON and reports explicit errors |
Google’s linear-regression material explains the weighted-input-plus-bias form. Our tiny fixture uses mathematical teaching data with no business interpretation. A real prediction problem needs meaningful training/evaluation data and a justified error measure; fitting this line proves none of that.
Run the local service and tests
Save all seven files together and inspect the five JavaScript files before running. This dependency-free code uses modern Node.js APIs; the actual run below used v25.2.1. Other runtime versions need their own test run.
node --test server.test.mjs measure.test.mjs
node measure.mjs > my-measurement.jsonThe measurement script starts a server on an available port at 127.0.0.1, runs its requests and closes it. Nine tests check fixed outputs, schema/numeric limits, HTTP errors and the measurement protocol. No network connection to a remote provider is made. Keep your new report alongside the supplied record: timings will differ, so compare the protocol, source hashes, counts and outputs—not identical milliseconds.
A completed run prints the full record even if requests fail. Exit code 0 means this fixture matched; 2 means a request failed or an expected value did not match; 1 means the measurement could not start or finish. None is production approval.
To try one request manually, run node server.mjs, copy its printed port into the command below, and stop the server with Ctrl+C afterwards:
curl http://127.0.0.1:PORT/predict -H "Content-Type: application/json" --data '{"inputs":[3,-1]}'Expected response: model version synthetic-linear-v1 and outputs [7,-1]. The model uses one scalar input per prediction, so token counts are not applicable—not zero-token LLM inference.
Test the interface, not only the formula
The server uses Node’s HTTP API. It accepts only POST requests to /predict with JSON containing an inputs array of 1–32 finite numbers in the range −1000 to 1000. Extra fields and numeric strings are rejected, not silently converted.
| Case | Expected HTTP result |
|---|---|
| Valid inputs [3,-1] | 200; ordered outputs [7,-1] |
| Malformed JSON, null input or extra fields | 400; invalid-input |
| Body larger than 4096 bytes | 413; body-too-large |
| GET on the prediction route | 405; use-post |
| Non-JSON content type | 415; use-json |
| Unknown route | 404; not-found |
These are teaching limits, not universal production defaults. The server has basic five-second request/socket limits, but the test suite does not establish timeout behaviour under every network condition. It has no authentication, TLS, persistent monitoring or multi-user protection. Keep it on loopback and do not expose it through a tunnel or deploy it as a public service.
Read the actual record, including its limits
Actual local observation: (10 September 2026 in Melbourne), using Node v25.2.1 on Apple M3 Pro, arm64, with 12 logical CPUs and 36 GiB of system memory. Client and server shared one process. The complete measured record contains every request and SHA-256 hashes of the five source/test files; this table reads directly from it.
Protocol inference-serving-measure-v2 compares the same inputs [3, −1, 0.5, 4] and expected outputs [7, −1, 2, 9]. Ten pairs alternate which mode runs first: balanced, not randomised. Both modes are synchronous with concurrency one and no retries. Two initial warm-up requests, single then grouped, are recorded separately; server startup is excluded, so this is not a cold-process benchmark.
| Pair / first mode | Four sequential requests, total ms | One four-input request, total ms | Exact matches: sequential / grouped |
|---|---|---|---|
| 1 / sequential | 3.523 | 0.673 | 4/4 / 4/4 |
| 2 / grouped | 2.756 | 0.638 | 4/4 / 4/4 |
| 3 / sequential | 1.735 | 0.415 | 4/4 / 4/4 |
| 4 / grouped | 1.630 | 0.691 | 4/4 / 4/4 |
| 5 / sequential | 1.476 | 0.375 | 4/4 / 4/4 |
| 6 / grouped | 1.490 | 0.348 | 4/4 / 4/4 |
| 7 / sequential | 1.646 | 0.349 | 4/4 / 4/4 |
| 8 / grouped | 1.326 | 0.384 | 4/4 / 4/4 |
| 9 / sequential | 1.377 | 0.345 | 4/4 / 4/4 |
| 10 / grouped | 1.363 | 0.332 | 4/4 / 4/4 |
Each mode matched 40/40 fixture values. Across warm-up and comparison there were 52 HTTP attempts, 85 scalar inputs and 0 failed requests. The request bodies totalled 857 bytes; completed response bodies totalled 2791 bytes. These are JSON-body counts, not network wire traffic or billable tokens.
Request timing includes local HTTP, body parsing and response-contract validation. Mode totals also include loop/bookkeeping overhead. We did not control host load, repeat on different machines or measure energy. Ten pairs on four mathematical inputs do not support a stable tail-latency estimate, generalisation claim or universal speed/cost winner. Do not convert this table into “batch inference is X times faster” marketing.
Both modes return the same simple arithmetic values. That is correctness against this fixture, not accuracy on an unseen business dataset. This grouped endpoint is not an asynchronous cloud batch service. Provider API spending was zero because none was called; total operating cost remains unmeasured.
A fast wrong response still fails the task
The measurement tests deliberately return one HTTP 503 after warm-up. The script still makes all 52 attempts; the sequential score becomes 39/40, not 39/39, and the failed slot stays in its original position. Another test returns HTTP 200 with wrong numeric outputs: transport errors stay at zero but both quality scores become 0/40. A 20 ms stalled-server test records a timeout. These are controlled failure tests, separate from the successful timing observation above.
Malformed JSON, unexpected model versions, extra fields and wrong output counts or types also produce explicit failure records. Redirects are refused. This protects the interpretation of this local experiment; it does not establish internet-facing service safety.
Create a usable inference handover record
- Record model/version, runtime, machine, date and exact input fixture; keep expected outputs outside any model input.
- Define timing boundaries, warm-up, request order, payload size, concurrency and repetitions. Keep failures and timeouts rather than dropping them from results.
- For an LLM extension, record actual input/output tokens, cache behaviour, price version/currency and retries. For total operating cost, also measure the relevant hardware allocation, energy and labour over the same boundary. These values are not supplied by this scalar lab: the JSON uses null, not invented zeros.
- Choose task-specific quality and safety criteria before testing. Compare errors as well as successful outputs.
- Record AI coding tool/model, proposed changes, manual interventions, rejected suggestions and the commands another builder should run.
- Have a second person reproduce the run and document differences. Do not claim this step is done until it happens.
This lab was drafted with AI coding assistance and tested automatically. The update added balanced ordering, failure-preserving quality counts and source hashes; it also fixed sparse-array validation for direct JavaScript calls. No independent human or recipient review is claimed. For your extension, ask your coding tool to add a failing request-validation test before changing the service. Review the diff; do not let it weaken the input contract to make the test pass.
If you already have broader delivery evidence, show MLAI Studio a build, its test results and operational limits. This exercise alone does not establish production readiness or eligibility for paid work. Current intake asks about contracting in Australia; confirm eligibility before assuming a New Zealand pathway.
A practical next step with MLAI
Show the service behind your AI demo
Have a build with reproducible tests and clear operating limits? Share your delivery evidence with MLAI Studio to be considered for scoped paid projects. Selection and matching are not guaranteed.
Apply with your delivery evidenceFor the decision-making layer around tools, try the PEAS agent-design lab. For discussion while learning, check relevant MLAI events.
Primary references checked 10 September 2026. The timings describe one local teaching run, not a vendor benchmark or measured customer benefit. Independent reproduction, production deployment and full operating-cost measurement have not been established.
Frequently Asked Questions
Must inference use previously unseen input?
Does inference mean the whole system stops learning?
Is the grouped request a cloud batch service?
Does zero API spending mean inference is free?
Disclaimer: This article provides general information and is not legal or technical advice. For official guidelines on the safe and responsible use of AI, please refer to the Australian Government’s Guidance for AI Adoption →
A practical next step with MLAI
Show the service behind your AI demo
Have a build with reproducible tests and clear operating limits? Share your delivery evidence with MLAI Studio to be considered for scoped paid projects. Selection and matching are not guaranteed.
Apply with your delivery evidence