PRODUCT ENGINEERING / CONNECTED BUSINESS SYSTEMS
OMNISTUGLOBALStart a project

Industry / AI and Data Products

AI products with a way to measure what gets better.

Bring retrieval, evaluation, product design and cost control into the same build, from a knowledge assistant to a workflow that can take approved actions.

Discuss your project β†—
Contents Β· AI and Data Products

Start with the work the product needs to support.

An impressive answer is a starting point. A useful product needs to find the right information, respect the person using it, show where an answer came from and make a bad result recoverable. Design those behaviors alongside the model interface, then measure them with cases from the work the product is meant to do.

The decisions behind the build.

Use case, model strategy and unit economics

Begin with the task, the people affected and the cost of a wrong answer. Separate a reviewed draft from a decision or action that changes another system. Model the actual request volume, input and output size, service charges, latency and operating work. Compare hosted and self-managed options against data handling, licences, available hardware and receiving capability. Choose fallback behavior for an unavailable or unsuitable model; no deployment model removes every dependency or privacy question.

Retrieval, and why the right context matters

Inspect the actual document types, tables, metadata and access rules. Test chunking that preserves meaning, keyword and semantic retrieval, reranking where useful, source freshness and deletion. Enforce the requesting user’s access before disclosing context to the model, with isolation across tenants and caches. Measure retrieval relevance separately from generated-answer quality. Retrieval and model choice both affect results; neither is universally the dominant factor or a guarantee of a grounded answer.

The evaluation harness alongside the product

Build representative cases with agreed expected behavior, source evidence, correctness criteria and appropriate refusals or abstentions. Combine deterministic tests for code and interfaces with task-specific automated scoring and calibrated human review. Repeat variable-output cases and retain versions, settings and outcome limits. Compare regressions when prompts, retrieval, models or dependencies change. Evaluation complements unit and integration tests; it does not replace them or establish that every future output is safe.

Safety, guardrails and adversarial input

Treat documents, web pages, files and model output as untrusted data. Separate application instructions from content, validate output for its destination and enforce authorization in execution code outside the model. Scope tool access to the actual caller, target and operation, with confirmed approval for consequential steps. Test direct and indirect injection, tenant isolation and denied actions using controlled data. Layered safeguards reduce risk, but no prompt or filter proves immunity to injection.

Cost control, routing and caching

Measure requests by model, feature, tenant and outcome, including retries, embeddings, retrieval, media processing and hosting where relevant. Compare candidate routing rules, cache behavior, context trimming and tenant limits against quality and latency. Cache keys must respect identity, permissions, source freshness and deletion; a fast hit that exposes another user’s answer is not an optimization. Agree budget ownership, alerts and a usable response when limits are reached.

Designing for the answer that is wrong

Show usable source references and make their limits visible. Distinguish measured confidence from generated expressions of certainty. Provide correction, abstention and review flows suited to the consequence, including a clear pending state before any action. Test whether people can understand and recover from a bad answer. A citation can be irrelevant or invented, so source display and explanatory text need validation rather than being treated as proof.

Launch and day two

Start with an appropriately bounded cohort and operational readiness checks. Collect only authorized telemetry with data minimization, redaction, retention, access control and deletion routes, not unrestricted raw input/output logging by default. Monitor quality, cost, failures and provider changes, and retain rollback or disable controls. Agree ongoing evaluation, receiving owners, incident response, support and update terms. No fixed delivery time or support period is established.

From a request to a checked answer.

A proposed implementation path. Confirm the actual states, failure behavior and acceptance evidence for your product.

Request

Identify the user, task, allowed data and action scope; validate inputs as untrusted.

Retrieve

Assemble permitted context with source lineage, relevance and freshness checks.

Generate

Call the selected model with bounded context, data permissions and measured cost.

Validate

Check output for the destination, factual evidence and any proposed tool operation.

Answer or seek approval

Show usable sources and limits; execute only through separately enforced authority and required approval.

Design for the parts that can fail.

Cache behavior is part of the permission model

A valid cache hit can avoid selected work, but may still require authentication, access checks, freshness checks and delivery cost. Test tenant separation and invalidation before reuse. Do not assume every model is billed per token or that a hit skips all other processing.

Tests for code and tests for answers

Software around the model still needs unit, integration and deterministic contract tests. Variable outputs also need representative graded cases, repeated runs and appropriate human review. Retain known limitations and compare model or retrieval updates on the same evidence rather than claim an evaluation set makes upgrades automatically safe.

Different operators need different workflows.

Assistants and chat products

Combine conversation state, permitted context, source references and correction or escalation. Test long conversations, missing evidence, access changes and cross-user separation before relying on a chat interface.

AI features inside SaaS

Fit the feature into actual tenancy, identity, billing, permissions and existing workflows. Verify which user’s data reaches retrieval, prompts, logs and caches, and enforce isolation across customers.

Document and knowledge systems

Preserve document structure, tables, provenance and document-level permissions through extraction and retrieval. Reflect source updates and deletion, and show when the evidence is missing or incomplete.

Agentic workflows

Model each proposed action, required authority, approval and resulting state. Limit tool arguments and destinations, handle partial failure and retain a reviewable action record without exposing unnecessary private data.

Generative media

Plan queues, latency, cost per output, moderation, consent, likeness and asset licences for the actual image, video or voice workflow. Provider access alone does not establish permission to generate or publish an output.

Code generation

Use isolated execution, restricted credentials, dependency review, tests and human review appropriate to the change. Passing generated tests does not establish secure behavior or rights in every generated dependency.

Vertical AI

Identify qualified domain reviewers, allowed uses, evidence standards, data restrictions and required human decisions for medical, legal, financial or other sensitive tasks. A software engagement is not a licence to practise.

AI infrastructure

Scope gateways, evaluation tools, data or vector services, developer documentation and operating interfaces. Test access control, compatibility, failure behavior, export and receiving maintenance, rather than infer reliability from a category label.

From discovery to a supported release.

Scoping against the cost of being wrong

Produce a written task definition, error consequences, permitted actions, data position and request economics. Identify users, reviewers, prohibited uses and the evidence that would make an ordinary software approach preferable.

Architecture around retrieval and trust boundaries

Document data ingestion, identity, retrieval, provider access, output handling and tool authorization. Define tenant isolation and cache boundaries, retention, deletion and the controls that remain outside model choice.

Build with the evaluation harness

Create the interfaces and evaluation cases together. Run code and integration tests plus graded task cases on relevant changes, recording source, model, prompt and retrieval versions and any human review required.

Adversarial testing and security validation

Test representative injection and data-isolation cases in an authorized environment, along with output handling and denied tool calls. Agree any independent security assessment, record findings and retest fixes. No supplied VAPT or compliance certification is asserted.

Cost instrumentation and tuning

Show cost by request, feature and tenant with quality and latency context. Test routing, caches, context sizes and limits against an approved budget and retain behavior when a provider is unavailable or a limit is reached.

Cohort launch and day two

Agree pilot access, privacy-conscious telemetry, support, incident response, rollback and receiving owners. Feed reviewed real failures into the evaluation set without treating unrestricted logging or automatic production updates as the default.

Connect the capabilities your product actually needs.

An evaluation set

Define representative tasks and expected behavior with known sources and review criteria. Test correctness, grounding, relevant refusal and legitimate task completion, and distinguish inconclusive cases from passes.

Retrieval and model selection together

Compare extraction, chunking, hybrid retrieval, freshness, permission filters and selected models against the same cases. Improve the bottleneck the evidence identifies, not the component a slogan says must matter most.

An injection boundary

Keep content separate from application authority and enforce access, destination and operation checks outside model output. Test the actual execution path as well as the visible answer.

Cost visible per feature

Measure provider and operating costs with retries, media or indexing where relevant, alongside successful task completion. Set budgets and alerts without silently changing the approved task or exposing tenant data in monitoring.

Output treated as untrusted

Use destination-appropriate escaping, parameterized queries, schema and argument validation, and constrained execution. A well-formed model response can still request an unauthorized action or carry incorrect facts.

Designed failure

Provide traceable evidence, uncertainty, correction, abstention and review. Make pending, denied, failed and completed actions distinct so a helpful-looking answer cannot hide a failed or unauthorized change.

Seven familiar product references, without assuming a ready base.

Use these names to describe the kind of interaction, not as affiliation, ownership, parity or proof that a platform is in production. Inspect any actual starting code, licences, provider dependencies and missing work before choosing adaptation over a new build.

ChatGPT-style assistant scope

Conversation, permission-aware knowledge retrieval, sources and review flows can be scoped around the actual product requirements. This is a category reference, not a claim of OpenAI affiliation or a ready deployed base.

Deepseek-style assistant scope

Assess the actual selected model or provider, licence, hosting, context handling and evaluation before choosing an assistant design. Brand resemblance does not establish rights, data location or equivalent capability.

Gemini-style multimodal scope

Scope the particular text, image or other supported inputs with provider and version-specific tests, retention and rights. No Google affiliation, parity or functioning demonstration is asserted.

Grok-style assistant scope

Define the required conversation, information freshness, grounding and safety controls. Test actual data sources and authorizations rather than assume a branded reference supplies them.

ReALM-style contextual scope

Assess the actual context signals, device access, user consent and model feasibility required for the workflow. The name is a planning reference, not an Apple product integration or shipped implementation.

HeyGen-style media scope

Scope generation queues, permitted likeness and voice use, moderation, review and per-output costs against the actual provider and rights. No media-provider affiliation or cloned production service is claimed.

bolt.new-style builder scope

Plan code generation, restricted execution, dependency review, preview and approved deployment with isolation and recovery. A named reference is not evidence of source ownership, code safety or a live builder.

Keep application authority outside generated content.

Application policy

Keep versioned application requirements and authorization in controlled code and configuration.

External content

Treat documents, pages, files and other users’ contributions as data, not permission to change goals or disclose information.

Actions

Enforce caller, target, operation and approval checks outside generation, and verify the actual resulting state.

Test hostile or misleading content, over-broad tool access, unsafe markup and unapproved irreversible operations with dummy data. These are generic product-security requirements, not a claim that a completed system is immune to attack.

Custom work when the standard workflow is not enough.

Retrieval systems

Build extraction, meaningful chunks, keyword/semantic search, source freshness, deletion and permission enforcement around the actual corpus. Measure retrieval and answer failures separately and retain evidence for debugging.

Agent architectures

Design bounded planning, tool interfaces and multi-step workflows with caller-scoped permissions, action-specific approvals and recovery from partial completion. Test actual writes and external disclosures with controlled state.

Evaluation platforms

Manage versioned cases, human and automated grading, repeated runs, regression comparisons and review queues. Keep test coverage and uncertainty visible instead of presenting a single score as universal quality.

Generative media pipelines

Connect approved generation providers, rights-cleared inputs, queues, safety review, storage and delivery with cost and latency controls. Confirm consent and publication scope before outputs leave the private workflow.

Classical machine learning

Compare forecasting, classification and recommendation approaches with a simple baseline and the task’s quality and cost needs. A trained model or ordinary rules may fit better than generation; decide from evidence.

Data foundations

Plan permitted collection, extraction, labelling, lineage, data quality and refresh or deletion. Validate source permissions and representative coverage before assuming better prompts can repair missing or unsuitable evidence.

Possible builds and the evidence to ask for.

These are illustrative requirements, not claimed completed projects or customer references. Genuine work can be assessed separately when its contribution and sharing rights are established.

Knowledge assistant with permission-aware retrieval

An illustrative scope combines authorized document search, tenant isolation, hybrid retrieval and inspectable sources. Acceptance should test access revocation, stale documents, incomplete context and a user asking for another user’s data.

Workflow agent with scoped tools

An illustrative scope proposes actions through bounded interfaces, verifies the caller’s permission and approval before execution and records the actual result. Test denial, partial completion and recovery with dummy records before connecting consequential systems.

Video generation with spend control

An illustrative scope combines rights-cleared inputs, queued generation, approval, per-tenant limits and cost reporting. Check cache identity and retention and ensure a job’s completion does not silently authorize publication. These are proposed builds, not verified client deployments.

Questions worth settling before implementation.

These notes stand on their own; they do not imply a published article, promised briefing pack or engineer-authored case study exists.

Testing probabilistic output alongside software

Use deterministic unit and integration tests for the surrounding code, plus repeated task evaluation and calibrated human review for output quality. A variable answer does not make all assertions or conventional tests inapplicable.

Finding the actual retrieval bottleneck

Measure whether relevant permitted material was found, whether it survived extraction and whether the answer used it correctly. Model selection, retrieval and product design need separate evidence rather than a universal priority ranking.

Authority at the action boundary

Validate the user, target, operation, arguments and approval in the execution path. A persuasive document, a model’s explanation or a clean final response does not establish that an action was authorized.

Cost per request as a product decision

Account for the complete successful task, including retries, retrieval, indexing, media and review where applicable. Compare quality and latency at the same time, and make limit behavior part of the product experience.

Bring the details that shape the first design.

What a wrong answer costs

Identify who relies on the output, whether a person reviews it and which failures or actions need a hard stop.

Where your data can go

List permitted providers, regions, hosting, retention, training use, access and onward disclosures, with the actual contractual requirements.

Volume and price

Bring realistic request, context, media and concurrency estimates and approved budget boundaries rather than a single token-cost assumption.

Existing systems

Map content, identity, permissions, business software, tool interfaces and the team that will receive and operate the product.

Questions about AI and Data Products.

Can we really launch in six working days?

No fixed launch duration is promised. Review the actual code, rights, integrations, data, evaluation and security work before agreeing a plan. A familiar chat interface does not make a new product’s evaluation set or acceptance automatic.

Do we own the source code?

Agree rights in the actual application code, third-party components, model weights, provider accounts, data and generated assets separately. Receiving source does not eliminate provider, hosting, hardware, licence or maintenance dependencies.

Which model should we use?

Compare candidates against representative cases, data handling, supported modalities, latency, quality, licences and complete cost. Routing and fallback can be useful but are not mandatory for every product. Test the actual provider versions rather than select from a generic ranking.

Can we run models on our own infrastructure?

Assess hardware, model licence, staffing, serving, monitoring, updates and total cost. Self-managed hosting can change external data flows but does not automatically remove training-data questions, security risk, third-party components or every external dependency.

How do we know if it is any good?

Agree representative cases and expected behavior, then combine code and integration tests with repeated output evaluation and appropriate human review. Keep correctness, grounding, refusal and legitimate task completion distinct, including known gaps and inconclusive results.

What about hallucination?

Retrieval, source checks, validation, abstention and human review can reduce or contain some errors, but do not guarantee their elimination. A citation or confident explanation may itself be wrong. Evaluate the consequence and verify the evidence before consequential use.

Is prompt injection a real risk for us?

Products that process untrusted user or external content should assess direct and indirect injection. Apply layered controls, caller-scoped authorization and destination-specific validation outside model output, and test the actual action path. Neither RAG nor a prompt alone proves protection.

How do we control cost?

Measure the actual request and completed task by feature and tenant. Then test routing, permitted caches, context size, queues and limits against quality, latency and approved spend. Include provider, indexing, hosting, retry and review costs where they apply.

Does our data train someone else's model?

Inspect the selected provider, exact service, settings, retention and contract, including subprocessors and any permitted training use. Self-hosting changes the questions but does not settle source-data rights, telemetry or every external flow. Put the required data handling in writing and verify the implementation.

Can the system take actions, not just answer?

Action-taking can be scoped with actual user authority, bounded interfaces, argument checks and confirmed approval for consequential steps. Test denial, partial failure and recovery. A model suggesting an action never establishes permission to execute it.

What happens when the model provider changes something?

Run the affected code and task evaluations, compare quality, cost, latency and data behavior, and use an approved rollout or rollback route. How long this takes depends on the change and evidence; no automatic safe upgrade or fixed review duration is promised.

What if we are not sure AI is the right answer?

Compare ordinary software, rules, search and classical ML against the actual task, uncertainty, data and costs. A discovery study can recommend a simpler option, more evidence or no build instead of forcing generation into the workflow.

Implementation and platform references.

Use documentation for the selected versions, devices, configuration and operating region. Project acceptance depends on the implementation, not a tool label.

Bring your workflows and operating requirements.

Share the users, business rules, current code and data, integrations, device/browser targets and receiving team. Agree the build boundaries, rights, acceptance evidence and operating responsibilities before implementation.

Discuss your project β†—