Seven Engineering Lessons from Running AI in Production
Abstract
This paper draws on the production use of an agentic platform across engineering, product, operations, and business teams. Teams use it to explore supply-chain data, trace delivery estimates, diagnose routing issues, investigate incidents, and answer operational questions. The platform connects models to company data and engineering tools, with identity and access checks, usage limits, audit logging, and monitoring. Since the work spans changing data and multiple systems, teams need to verify answers and understand how they were produced.
The paper presents seven lessons: keep policy and execution rules in deterministic code, validate inputs and outputs, limit retries and provide safe fallbacks or human handoff, record decisions for review, enforce permissions and budgets at runtime, deploy in stages with rollback, and separate model reasoning from system actions. It explains how to record useful decision details, such as model and tool versions, validation results, actions, latency, and outcomes, while limiting sensitive data. It also describes a release path from offline evaluation through shadow testing, canary release, measured rollout, and rollback.
These lessons are for teams introducing AI into workflows where answers can influence operational decisions. They offer a way to define what the model may do, what the system must verify, and what happens when a check fails. These controls help teams check outputs, recover from failures, and understand decisions.
- Published in
- Proceedings of the Raptors Conference, Vol. 1 (2026)
- Conference
- Raptors Conference 2026, 21 October 2026, online
- Accepted
- 2026-09-19
- DOI
- https://doi.org/10.68495/rc.2026.C054
- Presentation
- Slides (PDF)
- Licence
- © 2026Nirmal Kumar Jingar. Published under Creative Commons Attribution 4.0 International (CC BY 4.0).
- Publisher
- Hackathon Raptors, Community Interest Company 15557917, London, United Kingdom
- Permanent address
- https://proceedings.raptors.dev/jingar-seven-engineering-lessons-from
Open access. Every entry in the Proceedings is free to read, with no charge to authors and no charge to readers. Slides are served from this site, so they stay available at the paper's permanent address.
Cite this paper
Nirmal Kumar Jingar (2026). Seven Engineering Lessons from Running AI in Production. Proceedings of the Raptors Conference, Vol. 1 (2026). Hackathon Raptors, London. https://doi.org/10.68495/rc.2026.C054
@inproceedings{raptors2026-c054,
title = {{Seven Engineering Lessons from Running AI in Production}},
author = {Jingar, Nirmal Kumar},
booktitle = {Proceedings of the Raptors Conference},
volume = {1},
year = {2026},
publisher = {Hackathon Raptors},
address = {London, United Kingdom},
doi = {10.68495/rc.2026.C054},
url = {https://doi.org/10.68495/rc.2026.C054},
abstract = {This paper draws on the production use of an agentic platform across engineering, product, operations, and business teams. Teams use it to explore supply-chain data, trace delivery estimates, diagnose routing issues, investigate incidents, and answer operational questions. The platform connects models to company data and engineering tools, with identity and access checks, usage limits, audit logging, and monitoring. Since the work spans changing data and multiple systems, teams need to verify answers and understand how they were produced. The paper presents seven lessons: keep policy and execution rules in deterministic code, validate inputs and outputs, limit retries and provide safe fallbacks or human handoff, record decisions for review, enforce permissions and budgets at runtime, deploy in stages with rollback, and separate model reasoning from system actions. It explains how to record useful decision details, such as model and tool versions, validation results, actions, latency, and outcomes, while limiting sensitive data. It also describes a release path from offline evaluation through shadow testing, canary release, measured rollout, and rollback. These lessons are for teams introducing AI into workflows where answers can influence operational decisions. They offer a way to define what the model may do, what the system must verify, and what happens when a check fails. These controls help teams check outputs, recover from failures, and understand decisions.}
}
TY - CONF TI - Seven Engineering Lessons from Running AI in Production AU - Jingar, Nirmal Kumar T2 - Proceedings of the Raptors Conference VL - 1 PY - 2026 DA - 2026/09/19 PB - Hackathon Raptors CY - London, United Kingdom DO - 10.68495/rc.2026.C054 UR - https://doi.org/10.68495/rc.2026.C054 AB - This paper draws on the production use of an agentic platform across engineering, product, operations, and business teams. Teams use it to explore supply-chain data, trace delivery estimates, diagnose routing issues, investigate incidents, and answer operational questions. The platform connects models to company data and engineering tools, with identity and access checks, usage limits, audit logging, and monitoring. Since the work spans changing data and multiple systems, teams need to verify answers and understand how they were produced. The paper presents seven lessons: keep policy and execution rules in deterministic code, validate inputs and outputs, limit retries and provide safe fallbacks or human handoff, record decisions for review, enforce permissions and budgets at runtime, deploy in stages with rollback, and separate model reasoning from system actions. It explains how to record useful decision details, such as model and tool versions, validation results, actions, latency, and outcomes, while limiting sensitive data. It also describes a release path from offline evaluation through shadow testing, canary release, measured rollout, and rollback. These lessons are for teams introducing AI into workflows where answers can influence operational decisions. They offer a way to define what the model may do, what the system must verify, and what happens when a check fails. These controls help teams check outputs, recover from failures, and understand decisions. LA - en ER -