DZone Spotlight

Almost every backend eventually needs to run code on a schedule. Send the invoice at midnight. Retry the failed payment in five minutes. Generate the weekly report every Monday at 7 AM. Clean up expired sessions every hour. On one server, this is easy. You write a cron line and move on. The trouble starts when one server becomes ten. Now the same cron line lives on every box, so the invoice job fires ten times instead of once. Move the cron to a single “scheduler” box, and that box becomes a single point of failure. Every time you deploy new code, that process restarts, and if it crashes or the host dies, there is no second node to cover for it. Any job due during that downtime window silently never fires. A distributed job scheduler solves this. It runs jobs reliably across a fleet of machines, fires each job once even when nodes crash, and keeps working when parts of the system fail. This post walks through how to design one, the trade-offs at each step, and the mistakes that bite teams in production. What the Scheduler Has to Do Before drawing boxes, it helps to pin down the requirements. They split into two groups. Functional requirements: Run a job once at a specific time (a one-time job).Run a job on a repeating schedule, usually defined with cron (a recurring job).Support job dependencies, where job B runs only after job A succeeds.Retry a job automatically when it fails.Respect priority, so urgent jobs run before bulk jobs.Cancel or pause a job that is scheduled or already running. Non-functional requirements: Durability. Once the system accepts a job, it must not lose it, even if a node dies one second later.At-least-once execution. Every due job runs at least one time.Scale. The design should handle millions of jobs per day across many workers.Fault tolerance. A crashed worker must not block other jobs, and its work should be picked up by someone else. One requirement is worth calling out early. People often ask for “exactly-once” execution. In a distributed system, you cannot truly get it. What you can build is at-least-once delivery plus idempotent jobs, which together behave like exactly-once from the outside. More on that later. The Core Architecture The single most important idea in this design is to separate deciding when a job runs from actually running it. These are two different problems with different scaling needs, so they become two different components. A clean design has four parts: A scheduler that watches the clock and decides which jobs are due.A queue that holds ready-to-run jobs and hands them out.A pool of stateless workers that pull jobs and execute them.A datastore that holds job definitions and execution history, and acts as the source of truth. Why decouple the queue from the workers at all? Because load is bursty. At midnight, a thousand daily jobs may become due at the same second. If the scheduler called workers directly, that spike would hit them all at once. The queue absorbs the spike and lets workers drain it at a steady rate. It also lets you scale workers up and down without touching the scheduler. This is the same reason queues show up across system design, which I covered in detail in Role of Queues in System Design. Modeling Jobs in the Database The datastore is the source of truth, so the schema matters. A common approach uses two tables. One holds the recurring definition, the other holds individual runs. SQL CREATE TABLE jobs ( id BIGINT PRIMARY KEY, name TEXT NOT NULL, cron TEXT, -- null for one-time jobs payload JSONB, next_run_at TIMESTAMPTZ, -- when this job is next due enabled BOOLEAN DEFAULT TRUE ); CREATE TABLE job_runs ( id BIGINT PRIMARY KEY, -- unique id per run job_id BIGINT REFERENCES jobs(id), status TEXT NOT NULL, -- PENDING, RUNNING, SUCCEEDED, FAILED, DEAD attempt INT NOT NULL DEFAULT 1, scheduled_at TIMESTAMPTZ, started_at TIMESTAMPTZ, lease_until TIMESTAMPTZ ); CREATE INDEX idx_jobs_due ON jobs (next_run_at) WHERE enabled = TRUE; The partial index on next_run_at is the workhorse. The scheduler asks “which jobs are due now” many times per second, and this index keeps that query fast even with millions of rows. Each run moves through a small set of states. Drawing the state machine makes the retry and failure logic obvious. Defining Schedules With Cron Recurring jobs need a way to express “every day at 2:30 AM” or “every 15 minutes.” Cron is still the standard. A classic cron expression has five fields: Plain Text minute hour day-of-month month day-of-week 30 2 * * * -> 2:30 AM every day The Java world often uses Quartz cron, which adds a seconds field at the front and a year field at the end, giving six or seven fields. The two formats look similar but are not interchangeable, and mixing them up is a frequent source of jobs that never fire. The scheduler stores the cron string and computes a concrete next_run_at timestamp from it. After a run is enqueued, it computes the next one. This raises a real question: what happens if the scheduler was down for an hour and three runs were missed? This is the misfire problem. You generally pick one of two policies: Catch up. Run every missed occurrence in order. Correct for billing, expensive for everything else.Skip. Run only the next future occurrence and forget the missed ones. Right for jobs like cache refreshes where stale runs add no value. Make this an explicit setting per job. Teams that leave it implicit get surprised after the first outage. Picking Which Jobs to Run The scheduler needs to find due jobs and hand them off. There are three common ways to find them. Polling. Every second, query the database for jobs where next_run_at <= now(). Simple and reliable. The partial index keeps it cheap. The cost is a small delay, up to your poll interval.Timer wheel. Keep upcoming jobs in an in-memory structure sorted by time. Very precise and great for short delays, but you have to rebuild it from the database after a restart.Push. An external timing service fires an event when a job is due. Real-time, but now you depend on another moving part. For most systems, polling with a one-second interval is the right default. It is boring, and boring is good for a component you are trusting with billing runs. The harder problem is concurrency. If you run several scheduler instances for availability, they will all poll the same table at the same time. Without care, two of them pick the same job, and it runs twice. The clean fix in PostgreSQL is row locking with SKIP LOCKED: SQL SELECT id FROM jobs WHERE enabled = TRUE AND next_run_at <= now() ORDER BY next_run_at LIMIT 100 FOR UPDATE SKIP LOCKED; FOR UPDATE locks the rows this instance selects. SKIP LOCKED tells other instances to ignore locked rows and grab the next free ones instead. Many schedulers can now poll in parallel, each claiming a different batch, with no coordination service and no duplicate pickups. Airflow uses exactly this approach instead of a heavier consensus protocol, which is a good reminder that the simplest mechanism that meets the requirement usually wins. Why Exactly-Once Is a Myth Here is the scenario that breaks naive designs. A worker pulls a job, runs it successfully, and then crashes before it can tell the system “done.” The system still thinks the job is running. The lease expires, another worker picks it up, and the job runs a second time. You charged the card twice. You cannot delete this scenario. Networks drop messages and processes die at the worst moment. So you stop chasing exactly-once delivery and instead make the work safe to repeat. That means two things working together: At-least-once delivery. The system guarantees a due job runs at least one time, accepting that it may occasionally run more than once.Idempotent jobs. Running the same job twice has the same effect as running it once. The standard trick is an idempotency key built from stable identifiers, for example {job_id, run_id, attempt}, or a key tied to the business action like invoice_2026_06_charge. The worker records that key before committing side effects. If the same key shows up again, the worker sees the work is already done and acknowledges without repeating it. This is why each run gets its own unique id. A time-ordered id such as a Snowflake id or a ULID works well, because it is unique across the whole fleet without coordination and it sorts by creation time, which keeps the job_runs table naturally ordered. I explained the structure of these ids in How Snowflake IDs Work, and the deduplication pattern itself in Idempotent Receiver Pattern. There is one more subtle gap. The worker has to update the database and publish to the queue, and those are two systems. If it writes to the database and then dies before publishing, the job is lost. The transactional outbox pattern closes this gap by writing the job and an outbox row in one local transaction, then publishing from the outbox separately. I covered that in The Transactional Outbox Pattern. Coordinating at Scale A single scheduler instance has a throughput ceiling. Past a certain number of jobs per second, one process polling one database cannot keep up. There are two ways to grow. The first is leader election. You run several scheduler instances, but only one is active at a time. The others stand by and take over if the leader dies. A coordination service like etcd or ZooKeeper holds the leadership lock. This is simple to reason about, but the single active leader is still a throughput bottleneck. The second is sharding. You split the job space across many active schedulers. A simple scheme hashes the job id into one of N partitions, and each scheduler owns a set of partitions. Every job has exactly one owner, so there are no duplicate pickups, and throughput grows by adding schedulers. Consistent hashing makes it cheaper to add or remove schedulers without reshuffling everything. Sharding has one sharp edge. During a handover, while leases for a partition are changing hands, two schedulers can briefly believe they own the same partition. This is split brain. You do not try to make it impossible, because that is expensive. Instead, you let the worker-side idempotency check be the final safety net. If both schedulers enqueue the same run, the idempotency key means it still executes once. Google’s cron service takes a stricter route for its most sensitive launches. It writes the launch record to a quorum using Paxos before the job actually starts, so a failover cannot lose or double-fire it. For most teams, leases plus idempotency are enough, and full consensus is overkill. Detecting Failures and Recovering Workers crash. The scheduler has to notice and reassign their work, without stealing jobs from workers that are simply slow. The mechanism is a lease with a heartbeat. When a worker claims a run, it sets lease_until to a short time in the future, say 30 seconds. While the job runs, the worker periodically extends the lease. If the worker dies, it stops extending, the lease expires, and a recovery sweep moves the run back to PENDING so another worker can take it. SQL -- recovery sweep: reclaim runs whose lease has expired UPDATE job_runs SET status = 'PENDING' WHERE status = 'RUNNING' AND lease_until < now(); Two details make this robust. First, the lease timeout must be comfortably longer than a normal heartbeat interval, or a brief pause will cause a healthy job to be wrongly reclaimed. Second, you need protection against a zombie worker, one that froze on a long garbage collection pause, lost its lease, and then woke up and tried to finish writing results. A fencing token solves this. The reclaimed run gets a higher token, and the datastore rejects any write carrying an older token. I went deeper on time-bound ownership and fencing in The Lease Pattern in Distributed Systems. Retries Done Right A failed job should usually be retried, but retrying badly makes outages worse. If a downstream service is struggling and every failed job retries immediately, you pile on more load at the exact moment it can least handle it. The fix is exponential backoff with jitter. Each retry waits longer than the last, and a random jitter spreads the retries out so they do not all fire at the same instant. Plain Text attempt 1 fails -> wait ~1s attempt 2 fails -> wait ~2s attempt 3 fails -> wait ~4s attempt 4 fails -> wait ~8s (each wait randomized by +/- a few hundred ms) After a fixed number of attempts, stop. A job that keeps failing should not retry forever. Move it to a dead letter queue, a separate place for runs that exhausted their retries, and alert a human. The dead letter queue keeps a poisoned job from clogging the pipeline while preserving it for investigation. Operating the Thing A scheduler is infrastructure other teams depend on, so it has to be observable and controllable. For observability, track the metrics that tell you the system is healthy: Queue depth. A queue that keeps growing means workers cannot keep up.Scheduling lag, the gap between when a job was due and when it actually started.Run outcomes per minute, split by succeeded, failed, and dead.Lease reclaims, which spike when workers are crashing. For control, give operators real knobs. They should be able to pause a queue, drain a worker before a deploy so it finishes current jobs and takes no new ones, and replay a dead-lettered job after fixing the cause. Building these in from the start saves a lot of pain during the first incident. How Real Systems Approach This None of this is theoretical. The same building blocks show up across well-known tools, each making a different trade-off. Quartz. A mature Java scheduler. Multiple instances coordinate through a shared database using row locks, the same idea as the SKIP LOCKED approach above.Airflow. Orchestrates dependency graphs of tasks. Its scheduler uses database locks rather than a consensus protocol, favoring operational simplicity.Temporal. Models workflows as code and replays an append-only event history to recover state after a crash, which sidesteps a whole class of mid-task failure bugs.Celery. A popular task queue in Python, with a beat component that handles periodic scheduling.Kubernetes CronJobs. Run containerized jobs on a cron schedule inside a cluster, with configurable policies for missed runs and concurrency. See the Kubernetes CronJob docs.Google distributed cron. Writes launch state to a Paxos quorum before launching, so a leader failover never loses or doubles a run. The pattern across all of them is consistent. Decouple scheduling from execution, lean on the database or a quorum for coordination, accept at-least-once and make jobs idempotent, and design for failure as the normal case. Takeaways If you remember five things from this, make it these. Separate the decision of when a job runs from the work of running it. They scale differently.Do not chase exactly-once. Build at-least-once delivery and make every job idempotent.Use the database as a coordination primitive. SELECT ... FOR UPDATE SKIP LOCKED lets many schedulers poll safely.Use leases with heartbeats and fencing tokens to detect dead workers and reclaim their runs without double execution.Retry with exponential backoff and jitter, cap the attempts, and send the rest to a dead letter queue. A good scheduler is not clever. It is careful. It assumes nodes will die, messages will duplicate, and clocks will drift, and it keeps running anyway.

Then the search form grows, filters multiply, and nested criteria appear. Since using GET means placing the query inside the URI, a length limit problem emerges. Worse, placing sensitive query values in the URI increases the chance of exposure through access logs, browser history, proxies, and monitoring systems. Because the HTTP protocol does not forbid it, sending a body with GET may look like a way out, but building your design on behavior the standards leave undefined is not a recommended practice. Elasticsearch's GET-with-body search API is a well-known example, and Elastic's own documentation openly acknowledges the problem: "As a result, some HTTP servers allow it, and some—especially caching proxies—don't. [...] However, because GET with a request body is not universally supported, the search API also accepts POST requests." HTTP POST, on the other hand, carries the query in the request payload rather than the URI, which overcomes both the length limit and the data leakage problems. But POST is neither safe nor idempotent, since the protocol allows every invocation to change state on the server, and its response is not cached unless it carries explicit freshness information. This nature of POST also imposes a performance cost: results are recomputed and retransferred on every call, and a timed-out request cannot be safely retried. What is missing is clear: a method that is safe and idempotent like GET but carries content like POST. Until June 2026, HTTP did not have such a method in standardized form. The QUERY Method To address this need, the IETF introduced the QUERY method in RFC 10008. QUERY is the first new HTTP method since RFC 5789 was standardized in 2010. The core idea can be summarized as follows: a QUERY request asks the target resource to process the enclosed content in a safe and idempotent manner and to respond with the result. Everything else the RFC introduces either follows from this definition or builds practical machinery around it. Let's look at the key concepts one by one: Safe and Idempotent A QUERY is defined as a safe operation: it does not request a state change on the target resource. It can be retried, repeated, or restarted automatically without concern for partial side effects. This is the contract that separates it from POST. Meaning Comes From Content-Type RFC 10008 deliberately does not define a query language. The same endpoint may accept a JSON filter document, a form-encoded string, or any other query language defined by a media type; the media type of the request content defines how the server should interpret it. Servers are required to reject requests whose Content-Type is missing or inconsistent with the content. The RFC goes as far as forbidding content sniffing: a server is not allowed to infer a media type from the request content and use it to repair a missing or erroneous Content-Type. Explicitly Cacheable Unlike POST, QUERY introduces cacheability for body-carrying requests, with one crucial twist: the cache key must include the request content in addition to the URI, since two QUERY requests to the same URI with different bodies are different queries. Discovery via Accept-Query A server can advertise QUERY support with the Accept-Query response header, which lists the media types it accepts as query content. The Equivalent Resource A QUERY response may include a Location header pointing to a URI that represents the same query. A client can later re-fetch the result with a plain GET, no body required. The spec also gives 303 See Other a natural role for redirecting a query to a retrievable resource. The RFC's Security Considerations add one caveat here: when the query contains sensitive information that must not be logged, the URI assigned to such a resource should not include any sensitive portions of the original query content; otherwise, the exposure problem QUERY avoids would simply reappear one response later. Familiar Error Semantics The RFC recommends specific status codes for the failure cases: 400 when media type information is missing, 415 when the media type is not supported by the resource, and 422 when the content is well-formed but the query cannot be processed. A Decade in the Making The RFC had a long journey. The idea traces back to WebDAV's SEARCH method (RFC 5323, 2008), which demonstrated the demand for body-driven queries but remained confined to the XML-based WebDAV ecosystem. In 2021, the HTTP Working Group adopted the effort as a working group item, moving it from an individual proposal into the IETF standardization process. The method was later renamed from SEARCH to QUERY to avoid confusion with the existing WebDAV SEARCH method and to better reflect its purpose. The document was published as RFC 10008 in June 2026. Eleven years from the first draft to Proposed Standard is a useful reminder that even a seemingly simple addition to HTTP touches an enormous installed base and therefore receives extensive scrutiny. Where Ecosystem Support Stands Today As of July 2026, HTTP QUERY has completed the standardization phase with RFC 10008, but ecosystem adoption remains in its early stage. Many HTTP servers and proxies can forward QUERY requests without protocol changes, but native support across frameworks, browser APIs, caches, WAFs, and API tooling is still emerging. The primary barrier is no longer the protocol itself, but the large installed base of software that assumes a fixed set of HTTP methods. The Java ecosystem offers a useful snapshot of adoption in progress: Apache Tomcat A pull request adding QUERY support was merged on July 1, 2026 (apache/tomcat#1026). Support is available only in Tomcat 12 because it required Servlet API changes. Eclipse Jetty Eclipse Jetty has an open pull request (jetty/jetty.project#15316) implementing the core RFC 10008 semantics: method registration as safe and idempotent, the Accept-Query header, redirect behavior, and integration with compression and buffering handlers. It was initially aimed at Jetty 12.1 but has been retargeted to Jetty 13, aligning with a possible Jakarta Servlet 6.2 timeline. Jakarta Servlet There is an open issue (jakartaee/servlet#1068) proposing the addition of QUERY to the specification itself, so that HttpServlet gains first-class support and QUERY requests receive the same form parameter processing model currently defined for POST. This is arguably the most significant milestone for the broader Jakarta EE ecosystem, because it moves QUERY from container-specific support into the platform specification itself. Once Servlet defines QUERY, application servers such as WildFly, Payara, and Open Liberty can inherit support through their servlet containers as they move to the new specification level. As of this writing, none of them has shipped QUERY support ahead of the specification. What About Spring? Spring deserves its own section because of how request mapping is modeled. Spring MVC and WebFlux expose their annotation-based request mapping model through the RequestMethod enum, and that enum currently contains GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, and TRACE. There is no RequestMethod.QUERY, which means you cannot declaratively map a QUERY request through Spring's annotation-based programming model today. The available workarounds are awkward and bypass Spring's normal request-mapping model: declare a generic mapping and inspect request.getMethod() manually, or implement a custom RequestMappingHandlerMapping. Unlike the Servlet case, this is not primarily a container problem; it is primarily a framework API and abstraction problem. The Spring team is aware. A community pull request adding QUERY support (spring-projects/spring-framework#34993) has been open since before RFC 10008 was published. It supersedes a feature request that had remained open for nearly two years, and maintainers have indicated an intention to target Spring Framework 7.1, currently expected in November 2026. There is even a naming collision to solve first: the obvious convenience annotation @QueryMapping is already used by Spring for GraphQL. Why Quarkus Can Do It Today This is where an underappreciated property of HTTP pays off: the request method is simply a token defined by the HTTP grammar. A server does not need to have built-in knowledge of every method to parse it. Quarkus builds its HTTP layer on Netty and Vert.x, and neither requires the method to be one of a predefined set; the request can reach the routing layer without requiring special handling for QUERY. On top of that, Jakarta REST has had a standard extension point for custom methods since JAX-RS 1.0: the @HttpMethod meta-annotation, the same mechanism that has enabled JAX-RS applications to expose WebDAV methods like PROPFIND for years. Put the two together and RFC 10008-compatible QUERY endpoints in Quarkus require no framework changes; they can be enabled through a single Jakarta REST extension point: Java @HttpMethod("QUERY") @Documented @Target(ElementType.METHOD) @Retention(RetentionPolicy.RUNTIME) public @interface QUERY { } The remaining work is implementing RFC 10008 semantics at the application layer, which is precisely what the example project demonstrates. The Example: A Product Catalog You Can QUERY The demo repository is available on GitHub: hakdogan/http-query-method. It is a small Quarkus application exposing a product catalog at /products, deliberately compact, with only a handful of classes, but each RFC 10008 concept has a concrete counterpart in the code. One Query, Two Media Types The resource accepts the same logical filter in two representations, demonstrating that the query semantics are determined by the Content-Type, not the URI: Java @QUERY @Consumes(MediaType.APPLICATION_JSON) public Response query(ProductFilter filter) { ... } @QUERY @Consumes(MediaType.APPLICATION_FORM_URLENCODED) public Response queryForm(String body) { ... } So both of these work, and mean the same thing: Shell curl -i -X QUERY http://localhost:8080/products \ -H 'Content-Type: application/json' \ -d '{"category":"laptop","maxPrice":2000}' curl -X QUERY http://localhost:8080/products \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'category=laptop&maxPrice=2000' A request with an unsupported media type is rejected with 415, and a filter that is well-formed but self-contradictory, such as minPrice greater than maxPrice, returns 422. The second part is a design choice rather than an RFC requirement: Section 2.1 says 422 can be used when the content matches its media type, but the query cannot be processed due to its actual contents, and returning an empty result with 200 would be an equally valid reading. The demo treats the contradiction as a client error because an empty 200 response would be indistinguishable from a legitimately empty match, silently hiding what is almost certainly a bug in the caller. The Response Tells the Whole Story A successful QUERY comes back like this: Shell HTTP/1.1 200 OK Content-Type: application/json Accept-Query: application/json, application/x-www-form-urlencoded Location: http://localhost:8080/products?category=laptop&maxPrice=2000 Cache-Control: no-transform, max-age=60 ETag: "f675e29b" [{"category":"laptop","id":2,"name":"ThinkPad X1 Carbon","price":1899.00}, ...] Three headers carry the RFC's ideas: Accept-Query advertises which media types the resource accepts as query content. In the demo, it is added by a small response filter.Location points to the equivalent resource from Section 2.2 of the RFC: the same query expressed through the request URI. Fetch it with a plain GET, and you get the identical result, no body needed. One of the tests does exactly that round trip.Cache-Control and ETag make the cacheability promise concrete. The ETag is derived from the result, so repeating the query with If-None-Match returns 304 Not Modified without resending the result: Shell HTTP/1.1 304 Not Modified ETag: "f675e29b" This is the answer to "why not just POST": QUERY was designed to provide query semantics without giving up the cache-friendly properties associated with safe methods. Discovery Without Prior Knowledge How does a client discover that a resource supports QUERY? One OPTIONS request: Shell curl -i -X OPTIONS http://localhost:8080/products The response answers with two headers, one listing the methods the resource accepts and one listing the media types it accepts as query content: Shell HTTP/1.1 200 OK Allow: HEAD, QUERY, GET, OPTIONS Accept-Query: application/json, application/x-www-form-urlencoded In this case, Quarkus generated the Allow header automatically, including QUERY, simply because a resource method is bound to it. Proving Idempotency The demo's test suite covers the filtering logic, the media type handling, the error codes, the equivalent-resource round trip, the conditional request flow, and, fittingly for a method whose defining feature is repeatability, a test that repeats the same QUERY several times and verifies the operation remains safe and produces a consistent response. The key lesson from this example is not how QUERY was implemented, but why it was possible: the HTTP extension point already existed, and the framework did not need to invent a new abstraction. Conclusion QUERY is not a revolution; it is the standardization of a pattern that many systems have implemented through POST-based query endpoints for years. That is exactly why it matters. The gap between "works" and "works with the guarantees the protocol gives you" is where caching, idempotent retries, and better tooling become possible. Adoption is arriving unevenly: first in protocol implementations and servers, then in frameworks, gateways, and CDNs. But as the example shows, on a stack like Quarkus that treats the method as an extensible value rather than a hardcoded list, you do not have to wait to start experimenting. The protocol was ready for extension; the interesting question was whether the layers above it preserved that flexibility. The complete example, including all tests, is available on GitHub: hakdogan/http-query-method. References RFC 10008, The HTTP QUERY Method: https://www.rfc-editor.org/info/rfc10008/IETF Datatracker, document history: https://datatracker.ietf.org/doc/rfc10008/RFC 9110, HTTP Semantics: https://www.rfc-editor.org/info/rfc9110/RFC 4918, WebDAV: https://www.rfc-editor.org/info/rfc4918/RFC 5323, WebDAV SEARCH: https://www.rfc-editor.org/info/rfc5323/RFC 5789, PATCH: https://www.rfc-editor.org/info/rfc5789/

By Hüseyin Akdoğan CORE

For the past few years, large language models have felt unstoppable... Every few months, a bigger model arrived. Longer context. Better fluency. Fewer hallucinations. More parameters. The message was implicit but clear. Scale is intelligence. Then Meta released a paper in December 2025, co-authored by Yann LeCun, that quietly questioned the entire premise. Not by proposing a bigger language model. But by asking whether predicting words should ever have been the core of intelligence in the first place. The architecture introduced in that paper, Vision Language Joint Embedding Predictive Architecture or VL-JEPA, does not try to outdo LLMs at language. It sidesteps language entirely as the center of reasoning. That shift matters more than it first appears. The Hidden Assumption Behind LLMs Every LLM, no matter how large or refined, is built on one simple loop. Given what came before, predict what comes next. That loop has produced impressive results, but it also hides an assumption that we rarely question: that intelligence emerges from sequence completion. What LLMs Are Really Doing When an LLM writes: "My name is Ram Ghadiyaram," it does not hold the idea of a person introducing themselves. It does not represent identity. It does not reason about self-reference. It performs a statistical continuation: "My" -> "name" -> "is" -> "Ram" -> "Ghadiyaram" Each step is local. Each decision is linguistic. Meaning is never explicitly represented. It is inferred indirectly through correlations in text. This works surprisingly well, until you ask the system to reason, plan, or generalize beyond familiar language patterns. Why Scaling Language Hits a Ceiling Adding more data and parameters improves fluency, but it does not change the nature of the reasoning. LLMs still think left to right, depend on surface patterns, encode meaning implicitly rather than explicitly, and conflate knowing words with knowing the world. This is the core of LeCun's critique. Intelligence is not language. Language is a compression format humans use to communicate thoughts that already exist. The thought comes first. Recent research from OpenAI and DeepSeek (2024) documents saturation in LLM scaling laws around 10^26 compute FLOPs. This suggests we can no longer achieve intelligence gains through parameter volume alone. VL-JEPA's architectural shift promises to continue progress through representational depth instead. VL-JEPA Starts Where LLMs End VL-JEPA does not ask what word comes next. It asks, "what does this situation mean?" Instead of building intelligence on top of text generation, it builds text generation on top of understanding. Here, the key idea is to predict future representations, not future words. Those representations live in a semantic embedding space where images, video, text, and actions are mapped into the same conceptual coordinate system. Meaning becomes a first-class object. This architecture explains the human brain's convergence zones, where the visual cortex, auditory processing, and language areas integrate meaning without linguistic mediation. LeCun's self-supervised learning framework (2022,2024) provides the mathematical foundation for this biological alignment. LLM vs. VL-JEPA Architectures LLM architecture: VL-JEPA architecture: In LLMs, language is both the input and the reasoning medium. In VL-JEPA, language is just an interface. That distinction changes everything. Semantic Space Feels Closer to How Humans Think Humans do not reason word by word. We recognize situations, anticipate outcomes, and understand intent without narrating it internally. Semantic embeddings allow machines to do something similar. "A dog is running" "A puppy is playing" Different words. Same concept. Same region in semantic space. Contrast that with: "A dog is running" "A car is parked" Different concepts. Distant regions. No grammar required. No syntax rules. Just meaning. Performance and Cost Token generation is inherently sequential. Semantic reasoning is parallel and compact. A semantic embedding space processes information in parallel, potentially achieving 10x inference speedup compared to sequential token generation. Early benchmarks suggest VL-JEPA models with 7B parameters match LLM performance at 70B scale. That means faster inference, lower compute cost, and smaller models doing deeper reasoning. Robustness Prompt wording matters less. Paraphrasing does not confuse the system. Meaning stays stable across linguistic variations because the system reasons about the concept, not the words. Multimodal by Design Images are not translated into captions first. Videos are not reduced to text descriptions. Everything is understood in the same representational space. This unified approach eliminates the bottleneck of forcing all information through language, which has always been AI's weakest link for real-world understanding. LLMs Are Not Dead. They Are Demoted. The Meta paper does not argue for abandoning LLMs. Instead, it reassigns them. LLMs become translators between humans and machines, natural language renderers, and input normalization layers. They stop being the brain. They become the voice. World Models Are the Real Goal VL-JEPA is part of a larger idea called world modeling. A world model understands objects and their relationships, how actions change state, temporal continuity, and cause and effect. This is how humans plan. This is how animals navigate. This is how intelligence survives outside of text. Language alone cannot capture that. What Engineers Should Take Away This shift is architectural, not incremental. It suggests that future AI systems will separate reasoning from language, treat text as one modality among many, optimize for meaning rather than fluency, and scale understanding rather than vocabulary. Bigger prompts and longer context windows are not the endgame. Understanding is. The Production Timeline While VL-JEPA represents a fundamental shift, production adoption likely spans 18 to 24 months. Organizations should begin experimenting with semantic embedding frameworks now, treating this as the next major AI infrastructure transition. This shift does not mean abandoning LLMs, but reorganizing the AI stack. LLMs become specialized components rather than the foundation, similar to how Docker transformed container orchestration in 2012,2015. Engineers who understand both paradigms will lead this transition. Engineer Migration Strategy Start by understanding semantic embedding spaces in your existing toolkit. Experiment with multimodal models that blur the line between vision and language. Build prototypes that process information in parallel rather than sequentially. Most importantly, recognize that the bottleneck is no longer compute, but conceptual architecture. The engineers who thrive in this transition will be those comfortable with architectural change, not just model scaling. Conclusion This is not the end of large language models. It is the end of confusing eloquence with intelligence. The VL-JEPA paper marks a philosophical correction in AI. From words to worlds. From prediction to understanding. From language engines to cognitive systems. AI is not failing. It is finally asking the right question. Reference Meta AI Research, Yann LeCun et al. Vision Language Joint Embedding Predictive Architecture, December 11, 2025

By Ram Ghadiyaram CORE

Retrieval-augmented generation (RAG) reduces hallucinations in generative AI chatbots by grounding each response in retrieved source data instead of relying only on what the model learned during training. Before the model writes a reply, the system fetches relevant passages from a trusted knowledge store and passes them in as context. The model then answers from that evidence, which shrinks the room it has to invent facts. This article looks at why hallucinations happen at the token level, how a RAG pipeline counters them, and the engineering choices that decide whether grounding actually holds up in production. Why Generative AI Chatbots Hallucinate A large language model predicts the next token from statistical patterns, not from a fact store it can look up. Ask it about something outside its training data or about a recent change, and it still returns fluent, confident text, sometimes wrong. That confident-but-wrong output is a hallucination. Three causes show up most often in conversational AI systems: Knowledge gaps. The training corpus has a cutoff, so newer facts are missing.Ambiguous prompts. Vague input pushes the model to guess.Pattern completion. The decoder prefers plausible phrasing over accurate phrasing when both fit. For a customer-facing bot, the cost is concrete: invented pricing, fictional policies, or wrong API behavior, all delivered in the same tone as a correct answer. What Retrieval-Augmented Generation Actually Does RAG connects the model to an external knowledge base at query time. Rather than answering from parameters alone, the chatbot searches a document store first, pulls the closest matches, and injects them into the prompt. The pipeline runs in three stages: Retrieve: embed the user query and run a similarity search against a vector index.Augment: place the top passages into the prompt as grounding context.Generate: the model composes an answer constrained by that context. Because the output is tied to retrieved text, the system can also return citations pointing at the exact source. How RAG Reduces Hallucinations RAG targets the root cause: missing or stale context. Supplying current, relevant evidence narrows the space where the model has to improvise. Grounding in approved sources The model reads from your documents, so answers reflect your data rather than internet averages. A well-built pipeline also instructs the model to reply "not found" when retrieval returns nothing useful, instead of filling the gap with a guess. Fresh data without retraining You update the index, not the weights. New policies or product details become answerable the moment they are ingested, which removes a major source of dated, wrong replies. Traceable answers Each response can carry a reference back to its source passage. For regulated domains, that audit trail is often the difference between a system people use and one nobody trusts. A Minimal RAG Loop The core retrieval-then-generate step looks like this in pseudocode: Python def answer(query, index, llm): q_vec = embed(query) passages = index.search(q_vec, top_k=5) if not passages: return "I don't have that information." context = "\n".join(p.text for p in passages) prompt = f"Answer using only this context:\n{context}\n\nQ: {query}" return llm.generate(prompt) The top_k cutoff, the "only this context" instruction, and the empty-result fallback are small details that carry most of the anti-hallucination weight. Where RAG Pipelines Break Retrieval quality, not model size, is where most accuracy is won or lost. Common failure points: Bad chunking. Segments too large dilute relevance; too small and they lose meaning.Weak embeddings. A mismatched embedding model returns passages that look related but aren't.No reranking. Top-k by cosine similarity alone often buries the best passage below near-duplicates.Silent context overflow. When retrieved text exceeds the window, passages get truncated, and the model fills the gaps on its own. 2026 Patterns Worth Knowing A few shifts are changing how teams build these systems this year. Agentic RAG. Instead of one lookup, the chatbot plans multi-step retrieval, calling tools and querying several sources before answering. This handles compound questions a single search cannot. GraphRAG. Pairing a knowledge graph with vector search captures relationships between entities, which improves answers over connected or multi-hop data. Continuous evaluation. Automated grounding checks score every answer for faithfulness to its sources, catching regressions before users report them. As enterprise adoption grows, this kind of automated eval is moving from nice-to-have to default. Decision Factors Before You Build If you are weighing RAG for a production bot, the factors that matter most: Data freshness and cleanliness beat any single model choice.Chunking and overlap shape retrieval accuracy more than people expect.Guardrails: confidence thresholds and fallback responses so the bot declines rather than fabricates.An eval pipeline that measures grounding rate, not just fluency.Latency budget: retrieval adds round trips, so cache common queries. FAQ Does RAG remove hallucinations completely? No. It reduces them sharply, but noisy data or poor retrieval can still produce errors, which is why evaluation and guardrails stay necessary. Is RAG better than fine-tuning? For fresh, factual answers, RAG usually wins because you update data without retraining. Fine-tuning suits tone and format. Many systems use both. What data does a RAG chatbot need? A curated knowledge base: documentation, FAQs, policies, or product data, cleaned and chunked for retrieval. A Final Word Hallucination is the line between a chatbot demo and a system a team can put in front of real users. RAG addresses it directly by grounding generative AI chatbots in current evidence rather than hoping the weights remember. The hard part lives in retrieval quality and evaluation, not in the model alone.

I am not a developer. I want to say that upfront, because it changes everything about how you should read this. I run nenawow.com, a site that reviews AI tools and SEO software. Three years ago I had no SEO background and no coding background either. Last month I shipped six working tools that check AI visibility signals across any website, and they run without a database, without a backend, and without a single line of code I wrote myself. This is the story of how that actually happened. Why I Built This I kept hitting the same wall while testing AI visibility tools for my reviews. Most of them gave you one score and stopped there. A 64 out of 100 tells you something is wrong. It does not tell you what, where, or how long the fix will take. I wanted something different. Six focused tools, each checking one layer: crawler access, schema, content quality, llms.txt setup. Each one explaining the why behind every failed check, not just the fact that it failed. The problem was I could not write a single one of them myself. The Collaboration Model So I used Claude to architect and write every line of code. I described what I wanted each tool to check and why. Claude researched the technical approach, picked the architecture, and wrote the HTML, CSS, and JavaScript for all six tools plus the hub page that ties them together. My job was different. I tested everything. I deployed it. I caught what broke. I am the one who used the tools to write my own AI Visibility Benchmark article, scoring nine SEO publishers, so I know firsthand whether the output is trustworthy or not. That division of labor is the real subject of this article. Not "how I built a Cloudflare Worker." More like: how far can a non-developer get when the architecture decisions are sound and the testing discipline is real. First Architecture Decision: No Backend The first real decision was whether these tools needed a backend at all. Six tools that check live URLs need to fetch data from those URLs. A browser cannot do that directly because of CORS restrictions, the security rules that stop a webpage from freely calling other websites. The standard fix is a backend server that handles the fetch and passes the result back. A backend means a server to manage, a database maybe, ongoing hosting costs, and a lot more that can break. For one person running a site solo, that is a real cost. The architecture that got picked instead was a single Cloudflare Worker, a small script that runs at the edge and handles the CORS problem without any of that overhead. Here is the entire proxy. One file, around 35 lines, doing all the cross-origin work for every tool on the site. JavaScript export default { async fetch(request, env, ctx) { const corsHeaders = { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Methods': 'GET, POST, OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type', }; if (request.method === 'OPTIONS') { return new Response(null, { headers: corsHeaders }); } const url = new URL(request.url); const targetUrl = url.searchParams.get('url'); if (!targetUrl) { return new Response( JSON.stringify({ error: 'No URL provided' }), { headers: { ...corsHeaders, 'Content-Type': 'application/json', }, } ); } try { const response = await fetch(targetUrl, { headers: { 'User-Agent': 'Mozilla/5.0 (compatible; AIVisibilityChecker/1.0)', }, }); const text = await response.text(); return new Response( JSON.stringify({ content: text, status: response.status, }), { headers: { ...corsHeaders, 'Content-Type': 'application/json', }, } ); } catch (error) { return new Response( JSON.stringify({ error: error.message }), { status: 500, headers: { ...corsHeaders, 'Content-Type': 'application/json', }, } ); } }, }; That is the whole backend. It takes a URL as a query parameter, fetches it server-side where CORS does not apply, and hands the raw HTML back as JSON with permissive CORS headers attached. No routing, no auth, no state. Every tool calls it the same way. Here is the actual fetch from the Content Citability Grader, one of the six tools live on the site: JavaScript const PROXY = 'https://ai-visibility-proxy.nena46996.workers.dev/'; async function fetchViaProxy(url) { const res = await fetch( PROXY + '?url=' + encodeURIComponent(url) ); if (!res.ok) { throw new Error('Proxy request failed'); } const data = await res.json(); if (data.error) { throw new Error(data.error); } return data.content || ''; } One Worker. One URL. Every one of the six tools sends its fetch requests through it. That single decision is why "without a backend" in the title is not a marketing line. The Worker is the only server-side code in the entire system, and it does not know or care which of the six tools called it. What Almost Went Wrong: I Thought I Needed More Here is the part developers will recognize. Early on, I assumed something this complex needed a database, somewhere to store results, track usage, log scans. It did not. Look at what the Worker actually returns: raw HTML, nothing else. No scores, no analysis, no state. All of the actual intelligence, the regex pattern matching that checks for statistics, quotes, heading structure, FAQ schema, author bylines, lives entirely in the browser, in plain JavaScript running on the page itself. The Content Citability Grader scores four categories, Evidence, Structure, Authority, and AI Readability, by pattern-matching the fetched HTML client-side, the moment the response comes back. Nothing gets sent anywhere to be scored. Nothing gets saved after the tab closes. That split matters. The Worker's only job is solving CORS. The scoring logic, the actual product, runs for free in the visitor's own browser. No signup, no stored results, no database to maintain. That also matches a principle I hold for every tool I build: never hide information behind a signup, and never collect more than you need. I almost built more than the project needed. The stateless split between fetch and scoring made it unnecessary. Performance: What I Actually Measured I do not have lab-grade benchmarks here. What I have is real usage, from running the tools myself while building the AI Visibility Benchmark article, where I tested nine SEO publisher sites through all three relevant tools. Results typically came back in 2 to 4 seconds per scan. That held steady across all nine sites I tested, regardless of how large or complex the target page was. For a tool fetching live data from an external URL, parsing it, and scoring it in the browser, that is fast enough that nobody using it would call it slow. The full build, six tools plus the hub page, took three days. Ten-plus hours a day. Most of that time did not go into the Worker setup. It went into the scoring systems, getting the Content Citability Grader's four categories right, getting the Schema Checker's six schema types detecting correctly, and then connecting all six tools into one coherent workflow on the hub page. Mistakes: The Real Ones Two things broke during deployment that had nothing to do with the code itself. The first was Cloudflare. When I went to paste my Worker code into the project window, it would not take. No error message, nothing explained. The project window just kept showing old placeholder code, a default Hello World script, instead of accepting what I pasted. I tried seven times before it finally went through on the eighth attempt. I still do not know exactly what caused the first seven to fail. The second was WordPress, and this one took longer to figure out. I embedded each tool using a WPCode shortcode inside a Neve theme blank canvas page. After publishing, the live page showed two menus, the tool's own navigation duplicated alongside something from the Neve template. It looked broken even though the underlying tool worked fine. The fix turned out to be simple once I found it. Go back into the WPCode snippet, resave the exact same code with no changes, then go back to the page using that snippet and update it again. Preview after that, and the duplicate menu was gone. Nothing about the code changed. Something about how WordPress and Neve cached or registered the snippet did not sync correctly the first time around. Neither of these was a Worker problem or a JavaScript problem. They were platform quirks, the kind of thing no architecture diagram warns you about. The Worker code itself, once it finally deployed, has not needed a single change since. Its only failure path is the try/catch around the fetch, returning a 500 with an error message if the target site does not respond. That has been enough. Lessons: What I Would Tell Someone Doing This If I built this again, I would expect the platform friction before the code friction. The actual AI Visibility Checker, Schema Checker, and Content Citability Grader code worked close to correctly the first time, because the architecture was right from the start. What ate the most time was WordPress's caching behavior and Cloudflare's project window silently rejecting my first seven pastes. I would also tell anyone trying this without a developer background: the architecture decision matters more than your own coding skill. I could not have picked Cloudflare Workers over a traditional backend myself. I would not have known to ask the question. Getting that one decision right early is what let everything after it stay simple. The thing is, six tools sound like a big project. In practice, it was one architecture decision, repeated six times, with the real time going into getting each tool's scoring logic right rather than fighting infrastructure. Six tools. Three days. One Worker doing all the work nobody sees.

AI Accelerates Development, But Governance Determines Whether It Reaches Production Over the last two or three years, I’ve watched AI-assisted development move through distinct phases. First, general-purpose chat assistants like ChatGPT as a thinking partner. Then AI completions embedded in the IDE. Then, toward the end of 2025, the frontier agentic coding models arrived, and agentic development took a real leap in quality: harnesses, orchestrated agents, raw velocity. What came next is the part I care about most. Code volume exploded, and so did the number of problems riding along with it. I remember one of the strongest engineers on my team telling me, early this year, that he’d stopped reviewing lines of code. He was reviewing how the AI had reasoned. There was simply too much code to check any other way. That convinced many of us we needed governance. It started at the code level, moved up to design, and is now moving further upstream, into specs. AI has made software generation dramatically cheaper. In my opinion, it has not made understanding, verification, or accountability any cheaper, and that gap is where governance has to live. So governing AI-assisted development cannot mean reviewing every generated line twice or archiving every prompt. It means governing the changes AI introduces, whoever or whatever produced them. In practice, that rests on five pillars: Risk-based review gatesProvenance and traceabilityAutomated validationSecurity and dependency controlsAn updated Definition of Done 1. Treat AI as a Contributor, Not an Autonomous Developer An AI system can propose an implementation, challenge a specification, generate tests, investigate a production issue, or review a pull request. It cannot own the business outcome or accept the consequences of a bad release. That accountability remains with the engineering team. The useful division of labor is therefore not “AI writes, humans approve.” It is “AI performs work, while engineers own decisions.” This matters more as agents get better: a weak assistant exposes its uncertainty, but a strong one silently resolves an ambiguous requirement and hands back something that looks finished, so “looks right” quietly becomes “is right.” Governance must force important assumptions into the open before they disappear inside a diff. A workflow that separates spec, design, and implementation into explicit artifacts lets agents produce and critique each one, while humans keep the decisions that matter. 2. Build Review Gates That Focus on Risk, Not Authorship AI-generated changes should not automatically receive more scrutiny because of their author. Human-written code can contain security defects, copied dependencies, and misunderstood requirements too. The useful question is not “Did AI write this?” but “What could happen if this change is wrong?” A risk-based review policy considers factors such as: Whether the change touches authentication, authorization, payroll, financial data, or personal information.Whether it modifies a public API, database schema, infrastructure boundary, or permission model.Whether it introduces a dependency, changes a model or prompt, or affects an irreversible workflow.The blast radius, rollback difficulty, observability, and quality of automated evidence. Low-risk, well-bounded changes may be eligible for automated approval when CI, end-to-end tests, ownership rules, and agentic review all pass. High-risk changes should retain human gates even if every automated check is green. The safe way to get there is shadow mode: let the system record the decisions it would have made, compare them against human reviews, and only then hand it authority over a narrow, well-understood class of changes. This is human-in-the-loop by exception rather than by ritual. A human is pulled in when a security alert fires, a policy boundary is crossed, or the change falls outside the approved class. None of this leaves the pull request behind; it just asks the PR to carry more than a diff: the intent and design that make it reviewable at all. 3. Preserve Provenance From Prompt to Production Provenance is often confused with storing every conversation an engineer had with a model. That creates noise, privacy concerns, and a compliance archive that few reviewers can use. The objective is not prompt archaeology. It is decision traceability. For each material change, teams should be able to answer: Which ticket, incident, or business objective initiated it?Which specification and design decisions define the intended behavior?Which models, agents, skills, or reusable instructions materially influenced it?Which engineer or agent signed each gate, under which risk policy?Which validations ran, with what versions and results?Which artifact was deployed, where, and how can it be rolled back? That lives as structured metadata on the artifacts you already have: the issue, the spec, the commits, the pull request, the deployment record. Keep raw prompts only when reproducibility or a real regulatory need demands it. Treat reusable agent instructions, your organizational skills, rules, and domain context as version-controlled assets with owners and reviews. Otherwise, the same change will behave differently depending on someone’s undocumented personal prompt. 4. Expand Automated Validation Beyond Traditional Testing Unit, integration, and end-to-end tests remain essential. AI does not replace them; it increases their value because the volume and velocity of change are higher. But traditional tests only cover part of the new surface. An AI-assisted pipeline should also validate: Whether implementation behavior matches the approved specification and design.Whether generated code contains fabricated APIs, dead paths, unsafe defaults, or unnecessary complexity.Whether visual models, workflow definitions, prompts, and configuration are syntactically and semantically valid.Whether non-deterministic AI features meet evaluation thresholds on versioned datasets.Whether accessibility, performance, localization, migration, and rollback requirements are satisfied.Whether the change produces the telemetry required to detect failure after release. The key is independence: don’t let the same context author the code, write the tests, and declare success. Combine deterministic checks, a separate review agent, black-box behavior tests, and human judgment where it counts. For runtime AI features, tie prompt and model versions to offline evaluations and production traces, so a release gate means something and failures can be attributed to a version. 5. Strengthen Dependency and Security Governance AI-generated software must never bypass the existing security pipeline. It should pass the same static analysis, secret scanning, dependency review, and signing as any other change. Supply chain is the clearest example. Picture the everyday case: you open the laptop your company gave you, you hand a spec to an agent, and it gets to work. To hit the objective, it reaches for abstractions and pulls down whatever packages it decides it needs. Who checks that those packages aren’t the entry point of a supply chain attack? The agent doesn’t. It has no sense of a dependency’s history, its provenance, or whether that exact capability already exists inside your company. Security in an AI environment starts right there, at the first point of attack. So new dependencies have to be justified, pinned, scanned against vulnerability and license policy, and tracked in an SBOM, not taken on faith because the agent added them. But supply chain is almost the sophisticated worry. There are more obvious ones. Right now everyone in the organization is building software. Nobody wants to wait for the engineering department; they just want to solve their problem, automate a task, ship a small tool. The trouble is that most of these people have no way to tell whether the agent is making the right calls on authentication, authorization, or the access checks that decide who can see what. These were the bugs we used to see in junior teams. Now they show up everywhere, because everyone is generating code. Then there is corporate governance, which in many places has quietly collapsed. We gave agents open access so we could feed them all the context they needed, and that context sometimes included security tokens, even passwords. It is the same mistake as committing credentials in plaintext, or handing a third party the keys to company data. The difference is that this used to be harder, more controlled, or at least more obviously wrong: you didn’t casually point a test SaaS at the company data lake. Today people install a plugin in their harness and wire it to anything, source code and confidential data alike, whatever is convenient. The convenience of these tools is so seductive that nothing else seems to matter, as long as something gets automated. The fix isn’t exotic: corporate code, customer data, and credentials shouldn’t flow into unmanaged models or personal accounts in the first place, which means a managed environment, identity-based access, short-lived credentials, and agents that get the minimum permissions the task requires. And the stakes are not abstract. All of this maps directly to data breaches, with real impact on your customers and your company’s reputation. There has never been more surface to get this wrong, which means there has never been more risk. 6. Redefine "Definition of Done" for AI-Assisted Development A traditional Definition of Done quietly assumes the developer understands the code because they wrote it. AI breaks that assumption. When an agent produced the change, “it’s merged” no longer implies “someone understands it,” so the bar has to be explicit instead of taken for granted. It helps to separate two moments that are easy to blur. The Definition of Done applies first, on your own branch, before the change is ever proposed for integration. This is where the engineer who drove the agent confirms that its decisions are sound and that the change meets the requirements it had to meet. That does not mean rereading every line; it means having a plain, human-readable summary of how those requirements were validated and which decisions were taken along the way, clear enough that the engineer can judge whether the approach was right. That validation is the author’s job, and it does not move downstream to the merge gate. The review gates from earlier decide whether an already-validated change can be integrated, and that is the point where human attention becomes the exception rather than the rule. That summary never stands alone. It sits on top of the usual layers of security and behavior tests, unit, integration, and end-to-end, and together they are what provide real assurance: the tests prove the change behaves, the record of decisions explains why it was built that way. When a test fails, the failure and the reasoning behind it together are what let the engineer trace the root cause and improve the spec, so the next change starts from a better place. Not every change needs the same signer, but every change needs evidence proportionate to its risk. 7. Govern Model-Driven and AI-Assisted Delivery Together Enterprise applications are assembled from mixed artifacts now: source code, schemas, policy rules, prompts, retrieval sources, AI-generated components. And they keep moving that way. Low-code and visual platforms already let teams assemble behavior without hand-writing much code, and AI pushes it further, tailoring policies and workflows on demand by generating custom code behind the scenes. That generated code keeps the behavior deterministic instead of leaving it to a model at runtime, but it is still code no one wrote by hand or read line by line. Governing only the code you author creates blind spots. Picture a policy generated on demand that changes what customers can do in production, with no code review because no human opened a pull request. The behavior changed; the diff nobody read. A model upgrade can do the same to accuracy, cost, or latency while the application around it sits still. Governance should operate at the application-change level. One change record should tie every affected artifact — a generated policy, a visual workflow, a prompt — to the same intent, risk, evidence, and approval, so it travels through the same gates as hand-written code. This is where model-driven platforms earn their place: they provide the paved road, standard formats and common gates that make governance the default instead of an afterthought. The goal isn’t to slow teams down. It is to make the safe path the fastest path. Conclusion AI has already changed the economics of software creation. The teams that benefit most won’t be the ones that generate the most code, but the ones that turn that output into small, understandable, testable, and reversible changes. That takes a shift in mindset: one delivery system, not a parallel process for AI; risk classified, not authorship policed; humans escalated to by exception, not stationed in front of every low-risk change. What makes this urgent is what AI actually amplifies. It scales the output of every team, the ones with excellent judgment and the ones whose criteria are weaker, alike. Good governance is what raises the floor: it helps the second group decide better, not just ship faster, without slowing down the first. And it has to do this without killing exploration, because the real challenge here isn’t new. It’s the same tension we’ve always lived with: innovating versus standardizing to scale with quality. That balance is a judgment call, and it always will be. AI provides the speed. Governance provides the confidence to use it.

By Miguel Garcia CORE

Many engineering teams currently face a knowledge challenge. Information does exist; however, the information is distributed across various documentation formats such as design documents, runbooks, architectural notes, deployment guides, and incident reports. In general, a developer is aware of which services depend on each other (the Checkout Service depends upon the Payment API), the database or technology stack being used by the dependent services (the Payment API utilizes PostgreSQL), and who owns/operates the dependent service (Platform Team owns and operates the Payment API), however, these pieces of information typically reside in separate locations. A traditional search capability can locate documents with references to those terms. Using a retrieval-augmented generation (RAG) solution allows retrieval of relevant fragments/chunks based on contextually relevant keywords provided to the RAG model, which can then be passed along to a large language model (LLM). That strategy is effective for answering most types of questions. However, there are certain types of questions that are not simply about identifying the content of one document; they are about relating concepts. For instance: Plain Text Which team is owns the service that 'Checkout' relies upon? Relating all applicable data points is necessary when answering this type of question. To aid in the process of relating all applicable data points, a knowledge graph can become helpful. This article describes building a very basic knowledge graph from unformatted text using Python. As described above, keeping the example as simple as possible, but again, this is essentially how you would implement your own GraphRAG system: Identify entities from text, determine how those entities relate to each other, represent those relations as edges within a graph structure, and query that graph for relevant data prior to generating an answer. What We Are Building We will start with a few short engineering notes: Plain Text Checkout Service depends on Payment API. Payment API uses PostgreSQL. Platform Team owns Payment API. Recommendation Service calls Catalog API. Catalog API uses Elasticsearch. Search Team owns Catalog API. From those notes, we want to build a graph like this: Plain Text Checkout Service --DEPENDS_ON--> Payment API Payment API --USES--> PostgreSQL Platform Team --OWNS--> Payment API Recommendation Service --CALLS--> Catalog API Catalog API --USES--> Elasticsearch Search Team --OWNS--> Catalog API Once we have that structure, we can answer questions by traversing the graph instead of scanning raw text. Project Setup Create a new folder: Shell mkdir python-knowledge-graph cd python-knowledge-graph Create a requirements.txt file: Plain Text networkx==3.3 spacy==3.7.5 Install the dependencies: Shell pip install -r requirements.txt python -m spacy download en_core_web_sm We will use: spaCy for basic Natural Language Processing (NLP)NetworkX for building and querying the graph For this first example, we will not use a database. Keeping everything in memory makes the workflow easier to understand. Step 1: Identify The Input Documents Create a new file called build_graph.py. Python documents = ["Checkout Service depends on Payment API.", "Payment API uses PostgreSQL.", "Platform Team owns Payment API.", "Recommendation Service calls Catalog API.", "Catalog API uses Elasticsearch.", "Search Team owns Catalog API.", ] In a real-world deployment, the input document could have originated from a variety of sources (e.g., Markdown files, Confluence pages, GitHub repositories, service catalogs, incident reports). In this case, a couple of lines of example text should be sufficient to illustrate the concept. Step 2: Determine Relationship Triples Knowledge graphs typically store information in triple format (the subject, its relationship with another entity, and that other entity): Plain Text subject ---Relationship---> object An example would be: Plain Text Checkout Service --DEPENDS_ON--> Payment API In general, relationship detection in a large-scale application is often performed by a trained machine learning model. For demonstration purposes in this post, we'll utilize a simple rule-based detector to keep things straightforward. Add the following to build_graph.py: Python import re RELATION_PATTERNS = [ (r"(.+?) depends on (.+?)\.", "DEPENDS_ON"), (r"(.+?) uses (.+?)\.", "USES"), (r"(.+?) owns (.+?)\.", "OWNS"), (r"(.+?) calls (.+?)\.", "CALLS"), ] def extract_triples(text): triples = [] for pattern, relation in RELATION_PATTERNS: match = re.match(pattern, text, re.IGNORECASE) if match: subject = normalize_entity(match.group(1)) object_ = normalize_entity(match.group(2)) triples.append((subject, relation, object_)) return triples def normalize_entity(value): return value.strip() This function is intentionally simple. It looks for a small set of verbs and converts each sentence into a graph-friendly structure. Try it: Python for doc in documents: print(extract_triples(doc)) Expected output: Plain Text [('Checkout Service', 'DEPENDS_ON', 'Payment API')] [('Payment API', 'USES', 'PostgreSQL')] [('Platform Team', 'OWNS', 'Payment API')] [('Recommendation Service', 'CALLS', 'Catalog API')] [('Catalog API', 'USES', 'Elasticsearch')] [('Search Team', 'OWNS', 'Catalog API')] This is the first useful step. We have converted unstructured text into structured facts. Step 3: Create a Graph With NetworkX We can now add those triplets into a directed graph. Python import networkx as nx def build_knowledge_graph(documents): graph = nx.DiGraph() for doc in documents: triples = extract_triples(doc) for subject, relation, object_ in triples: graph.add_node(subject) graph.add_node(object_) graph.add_edge(subject, object_, relation=relation, source_text=doc) return graph The use of a directed graph makes sense when dealing with relations that are directional. This: Plain Text Checkout Service --DEPENDS_ON--> Payment API does not mean the same thing as this: Python Payment API --DEPENDS_ON--> Checkout Service Direction matters for dependency analysis, ownership lookup, impact analysis, and retrieval. Step 4: Print the Graph Add a helper function: Python def print_graph(graph): for source, target, data in graph.edges(data=True): relation = data["relation"] print(f"{source} --{relation}--> {target}") Now run the full flow: Python if __name__ == "__main__": graph = build_knowledge_graph(documents) print_graph(graph) Output: Plain Text Checkout Service --DEPENDS_ON--> Payment API Payment API --USES--> PostgreSQL Platform Team --OWNS--> Payment API Recommendation Service --CALLS--> Catalog API Catalog API --USES--> Elasticsearch Search Team --OWNS--> Catalog API At this point, we have a working knowledge graph. It is small, but it already gives us something normal keyword search does not: explicit relationships. Step 5: Query the Graph Let’s answer a practical question: Plain Text Who owns the API that Checkout Service depends on? That requires two hops: Plain Text Checkout Service -> Payment API -> Platform Team The first hop finds the dependency. The second hop finds the owner. Add this function: Python def find_owner_of_dependency(graph, service_name): results = [] for dependency in graph.successors(service_name): edge_data = graph.get_edge_data(service_name, dependency) if edge_data["relation"] != "DEPENDS_ON": continue for possible_owner in graph.predecessors(dependency): owner_edge = graph.get_edge_data(possible_owner, dependency) if owner_edge["relation"] == "OWNS": results.append( { "service": service_name, "dependency": dependency, "owner": possible_owner, } ) return results Call it: Python owners = find_owner_of_dependency(graph, "Checkout Service") for item in owners: print( f"{item['owner']} owns {item['dependency']}, " f"which is used by {item['service']}." ) Output: Plain Text Platform Team owns Payment API, which is used by Checkout Service. This is a simple example, but it shows the main value of graph-based retrieval. We did not search for similar text. We followed relationships. Step 6. Save the Graph When building a small prototype, you can often save the graph as JSON. Here’s how to do that. Python import json def export_graph(graph, output_path): data = { "nodes": list(graph.nodes()), "edges": [ { "source": source, "target": target, "relation": edge_data["relation"], "source_text": edge_data["source_text"], } for source, target, edge_data in graph.edges(data=True) ], } with open(output_path, "w", encoding="utf-8") as file: json.dump(data, file, indent=2) export_graph(graph, "graph.json") The output looks like this: JSON { "nodes": [ "Checkout Service", "Payment API", "PostgreSQL", "Platform Team", "Recommendation Service", "Catalog API", "Elasticsearch", "Search Team" ], "edges": [ { "source": "Checkout Service", "target": "Payment API", "relation": "DEPENDS_ON", "source_text": "Checkout Service depends on Payment API." } ] } Keeping the original source_text when saving the graph is important. This is because if you were to plug this graph into either a RAG or GraphRAG pipeline, there needs to be some way for the LLM to see not just the relationship within the graph but also have access to the actual supporting text for each one. Where spaCy Fits In Regular expressions were used in the previous example because of how simple your examples were (you know exactly what will appear). However, most real-world documentation is not as straightforward. This is why we use spaCy. It can find all types of named entities; e.g., organization, product, person, etc. Below is a small sample: Python import spacy nlp = spacy.load("en_core_web_sm") text = "Platform Team owns Payment API, which uses PostgreSQL." doc = nlp(text) for entity in doc.ents: print(entity.text, entity.label_) As with many things with regard to NLP, depending upon your specific model and the input data you provide, spaCy may automatically find some entities. However, generally speaking, you will need to either develop custom rules or train your own model to extract certain types of entities that pertain to specific domains of interest (e.g., engineering-related terms such as internal services, APIs, teams, etc.). In practice, one common method to take advantage of this is to use a combination of approaches: Plain Text Use spaCy for general entity extraction. Use rule-based patterns for known engineering relationships. Use an LLM only when the relationship cannot be extracted reliably with simpler methods. Using a combination of these methods provides a way to keep costs and complexity at reasonable levels. How This Connects to GraphRAG Pipeline GraphRAG (graph retrieval-augmented generation) uses a graph during the retrieval phase prior to generating an answer using an LLM. Markdown Documents | Entity and relationship extraction | Knowledge graph | Graph traversal | Supporting text | Large Language Model | Answer This article implements the graph construction and traversal steps only; LLM integration is outside the scope of this example. The graph is used instead of traditional RAG in many cases. However, it is used most often when the question relates to entities that have some form of relationship to one another. Good fit: Plain Text Which services are indirectly affected by a Payment API outage? Usually not worth the extra complexity: Plain Text What is the timeout value for Payment API? For simple fact lookup, vector search may be enough. For dependency, ownership, impact analysis, and multi-hop questions, graph retrieval can add real value. Complete Example Here is the complete script: Python import json import re import networkx as nx documents = [ "Checkout Service depends on Payment API.", "Payment API uses PostgreSQL.", "Platform Team owns Payment API.", "Recommendation Service calls Catalog API.", "Catalog API uses Elasticsearch.", "Search Team owns Catalog API.", ] RELATION_PATTERNS = [ (r"(.+?) depends on (.+?)\.", "DEPENDS_ON"), (r"(.+?) uses (.+?)\.", "USES"), (r"(.+?) owns (.+?)\.", "OWNS"), (r"(.+?) calls (.+?)\.", "CALLS"), ] def normalize_entity(value): return value.strip() def extract_triples(text): triples = [] for pattern, relation in RELATION_PATTERNS: match = re.match(pattern, text, re.IGNORECASE) if match: subject = normalize_entity(match.group(1)) object_ = normalize_entity(match.group(2)) triples.append((subject, relation, object_)) return triples def build_knowledge_graph(documents): graph = nx.DiGraph() for doc in documents: triples = extract_triples(doc) for subject, relation, object_ in triples: graph.add_node(subject) graph.add_node(object_) graph.add_edge(subject, object_, relation=relation, source_text=doc) return graph def find_owner_of_dependency(graph, service_name): results = [] for dependency in graph.successors(service_name): edge_data = graph.get_edge_data(service_name, dependency) if edge_data["relation"] != "DEPENDS_ON": continue for possible_owner in graph.predecessors(dependency): owner_edge = graph.get_edge_data(possible_owner, dependency) if owner_edge["relation"] == "OWNS": results.append( { "service": service_name, "dependency": dependency, "owner": possible_owner, } ) return results def export_graph(graph, output_path): data = { "nodes": list(graph.nodes()), "edges": [ { "source": source, "target": target, "relation": edge_data["relation"], "source_text": edge_data["source_text"], } for source, target, edge_data in graph.edges(data=True) ], } with open(output_path, "w", encoding="utf-8") as file: json.dump(data, file, indent=2) if __name__ == "__main__": graph = build_knowledge_graph(documents) for source, target, data in graph.edges(data=True): print(f"{source} --{data['relation']}--> {target}") owners = find_owner_of_dependency(graph, "Checkout Service") for item in owners: print( f"{item['owner']} owns {item['dependency']}, " f"which is used by {item['service']}." ) export_graph(graph, "graph.json") Run it: Shell python build_graph.py Production Considerations This example is intentionally small. In a real system, the hardest part is not creating the graph. It is keeping the graph clean. A few things matter quickly: Entity normalization: Payment API, payment-api, and Payments API may all refer to the same system.Relationship quality: Bad relationships are worse than missing relationships because they lead retrieval in the wrong direction.Source tracking: Every edge should preserve where it came from. This helps with debugging, trust, and answer citation.Incremental updates: Rebuilding the entire graph every time a document changes is usually wasteful.Storage choice: NetworkX is excellent for local prototypes. For larger graphs, use a graph database such as Neo4j or another persistent graph store. Key Takeaways A knowledge graph is a practical way to represent relationships hidden inside documents. You do not need a complex architecture to get started. A small Python script can extract triples, build a graph, and answer multi-hop questions. Graph-based retrieval is most useful when the answer depends on connections between entities. It is less useful for simple lookup questions where traditional search already works well. The foundation of a good GraphRAG system is not the LLM prompt. It is the quality of the entities, relationships, and supporting evidence in the graph. Try It Yourself Add these two documents: Plain Text Checkout Service runs on Kubernetes. Platform Team manages Kubernetes. Then add a new relationship type called RUNS_ON. Update the query function to answer: Plain Text Who manages the platform that Checkout Service runs on? This small exercise will help you see why graph traversal becomes useful as relationships grow.

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen. Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel. In this blueprint, we will build a secure multi-tenant execution environment. We will containerize the agent runtime using Docker, provision a GKE cluster with Axion nodes, isolate the network, and orchestrate the execution layer using a robust Java backend. Step 1: Containerizing the Agent Runtime The first step is establishing a baseline execution environment. We want this Docker image to be as lightweight as possible to reduce the attack surface, while containing the necessary runtimes for the LLM to execute its logic. Dockerfile # Use a minimal Alpine base image to reduce attack surface FROM python:3.11-alpine # Create a non-root user for execution RUN addgroup -S agentgroup && adduser -S agentuser -G agentgroup WORKDIR /sandbox # Copy the execution wrapper script COPY --chown=agentuser:agentgroup execute_payload.py /sandbox/ # Enforce non-root execution USER agentuser # Prevent Python from writing pyc files and buffering stdout ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 CMD ["python", "execute_payload.py"] To make this functional, we need an entrypoint script that safely reads the LLM-generated code from an injected environment variable or a mounted volume, executes it, and captures the output. Here is a simplified execute_payload.py implementation: Python import os import sys import traceback def main(): # In a production environment, this payload might be injected via # a Kubernetes Secret or a secure sidecar proxy. encoded_payload = os.environ.get("AGENT_PAYLOAD", "") if not encoded_payload: print("Error: No payload provided.") sys.exit(1) try: # Execute the untrusted code within this isolated process # Security constraints are handled by the container and gVisor layers exec(encoded_payload, {"__builtins__": __builtins__}, {}) except Exception as e: print(f"Execution Error: {str(e)}") traceback.print_exc() sys.exit(1) if __name__ == "__main__": main() Even if a malicious script breaks out of the Python runtime, it will find itself as an unprivileged user inside a minimal Alpine container. Step 2: Provisioning GKE With Axion and Agent Sandbox Google Axion (N4A) processors provide excellent performance per watt, making them ideal for running hundreds of concurrent, lightweight agent tasks. We will create a cluster and explicitly enable the sandbox feature. Shell # Create the GKE cluster with Sandbox enabled gcloud container clusters create agent-sandbox-cluster \ --region us-east4 \ --enable-sandbox \ --sandbox type=gvisor \ --release-channel regular # Create a dedicated node pool using Axion N4A instances gcloud container node-pools create axion-agent-pool \ --cluster agent-sandbox-cluster \ --region us-east4 \ --machine-type n4a-standard-4 \ --num-nodes 3 \ --node-labels dedicated=untrusted-agents \ --tags untrusted-workload Applying node labels ensures that trusted core microservices do not accidentally end up on the same physical infrastructure as untrusted agent execution environments. Step 3: Enforcing Network Isolation Compute isolation is useless if the untrusted code can scan your internal network or exfiltrate data to the public internet. We must deploy a strict NetworkPolicy to default-deny all egress traffic from our sandboxed namespace. YAML apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-agent-egress namespace: isolated-agents spec: podSelector: matchLabels: app: agent-executor policyTypes: - Egress egress: # Only allow DNS resolution - ports: - port: 53 protocol: UDP - port: 53 protocol: TCP # Allow outbound only to a specific internal API gateway if needed # - to: # - ipBlock: # cidr: 10.0.0.50/32 Step 4: Deploying the Sandboxed Workload With the network secured, we define the Kubernetes deployment. By setting the runtimeClassName to gvisor, Kubernetes routes the container lifecycle through the GKE Agent Sandbox rather than the standard container runtime. YAML apiVersion: apps/v1 kind: Pod metadata: generateName: dynamic-agent-task- namespace: isolated-agents labels: app: agent-executor spec: # Instruct GKE to use the Agent Sandbox (gVisor) runtimeClassName: gvisor # Ensure these pods only land on our Axion node pool nodeSelector: dedicated: untrusted-agents restartPolicy: Never containers: - name: execution-environment image: your-registry/agent-runtime:v1.0.0 env: - name: AGENT_PAYLOAD valueFrom: secretKeyRef: name: task-payload-secret key: payload # Drop all unnecessary Linux capabilities securityContext: runAsUser: 1000 runAsNonRoot: true allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: - ALL resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" volumeMounts: - name: temp-storage mountPath: /tmp volumes: - name: temp-storage emptyDir: {} Step 5: Orchestrating the Execution via Java Spring Boot To bring this architecture together, the control plane must dynamically spin up these sandboxed pods whenever an AI agent decides it needs to run code. In a modern distributed system, this is typically handled by a core backend microservice. Using the Fabric8 Kubernetes Client in a Java Spring Boot application provides a highly resilient way to orchestrate these ephemeral workloads programmatically. Java import io.fabric8.kubernetes.api.model.Pod; import io.fabric8.kubernetes.client.KubernetesClient; import org.springframework.stereotype.Service; @Service public class AgentOrchestratorService { private final KubernetesClient kubernetesClient; public AgentOrchestratorService(KubernetesClient kubernetesClient) { this.kubernetesClient = kubernetesClient; } public String executeUntrustedCode(String tenantId, String pythonCode) { // 1. Create a Kubernetes Secret containing the code payload String secretName = createPayloadSecret(tenantId, pythonCode); // 2. Load the sandbox Pod template and inject the specific payload secret Pod sandboxedPod = kubernetesClient.pods() .inNamespace("isolated-agents") .load(getClass().getResourceAsStream("/k8s/agent-pod-template.yaml")) .item(); // 3. Launch the pod dynamically via the API server Pod runningPod = kubernetesClient.pods() .inNamespace("isolated-agents") .create(sandboxedPod); // 4. Await completion and extract the logs safely kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .waitUntilCondition(pod -> pod.getStatus().getPhase().equals("Succeeded") || pod.getStatus().getPhase().equals("Failed"), 30, java.util.concurrent.TimeUnit.SECONDS); String executionLogs = kubernetesClient.pods() .inNamespace("isolated-agents") .withName(runningPod.getMetadata().getName()) .getLog(); // 5. Clean up the ephemeral resources kubernetesClient.pods().delete(runningPod); kubernetesClient.secrets().withName(secretName).delete(); return executionLogs; } } The Defense in Depth Strategy This architecture relies on a strict defense in depth model. If an LLM hallucinates a malicious payload or a user deliberately attempts prompt injection to compromise the platform, the attacker faces multiple independent barriers. The code executes as a non-root user in a minimal Alpine environment with a read-only filesystem. Network access is completely blocked by native Kubernetes policies. Finally, any attempt to exploit kernel vulnerabilities is intercepted by the gvisor runtime boundary running on dedicated Axion hardware. By combining these layers, engineering teams can build and scale trustworthy Agentic AI platforms without risking the integrity of their core cloud infrastructure.

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we'd agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas. What it didn't account for is that each replica needed almost three minutes just to pull a 14GB checkpoint and warm up CUDA kernels before it could answer a single request. The load balancer, seeing new pods report as running, immediately started routing traffic to them. For three minutes, a chunk of our users got 504s while perfectly healthy-looking pods sat there loading a model into memory. Nobody on the infra side had touched Docker that day. The incident wasn't a Docker bug. We assumed that container orchestration designed for web services would function the same way for processes that take minutes to become useful, rather than those that operate in milliseconds. Why LLM Containers Break the Usual Assumptions Packaging an LLM serving stack in Docker still makes sense for the same reason it always has; CUDA versions, driver compatibility, and Python ABI mismatches are miserable to manage across a fleet without a frozen artifact. But an LLM container carries baggage that a typical inference service doesn't. The weights are tens of gigabytes, not a few hundred megabytes. GPU memory is a single shared pool that one greedy container can quietly exhaust for everyone else on the box. And “ready” doesn't mean “process started”; it means the model is resident in VRAM and the CUDA graph is warmed, which can take minutes on a cold node pulling weights from object storage over the network. The Mistakes, in Order Our first version baked the model weights directly into the image, because it felt simpler: one artifact, one pull, done. In practice, it meant a 16GB image, painfully slow CI pushes, and a registry bill nobody wanted to look at. Worse, every time we bumped into a new fine-tuned checkpoint, we rebuilt and repushed the entire layer regardless of caching, because the COPY step touching gigabytes of weight files invalidates everything below it. Unlike a typical ML inference image, there's no meaningful caching win here at all; the layer is simply too big to ever be a cache hit across versions. We moved weights out to a mounted volume, fetched at container start from object storage, and never looked back. Second mistake, and this one actually cost us a production incident: we ran the container with Docker's default shared memory size. vLLM, which we used for serving, spins up worker processes that talk to each other over shared memory even on a single GPU. With the default 64MB /dev/shm, those workers would crash with cryptic bus errors under any real concurrency. The fix was almost embarrassingly small: Shell docker run --gpus all \ --shm-size=2g \ -e MODEL=mistralai/Mistral-7B-Instruct-v0.2 \ -e GPU_MEMORY_UTILIZATION=0.85 \ -e MAX_MODEL_LEN=8192 \ -p 8000:8000 \ llm-serve:latest The third mistake was more subtle and took longer to diagnose. vLLM's continuous batching reserves a large slice of GPU memory upfront for the KV cache, controlled by gpu_memory_utilization. We'd set that fraction high to maximize throughput, then bin-packed two replicas onto the same GPU to save cost. Under normal traffic, fine. During a burst of unusually long-context requests, such as someone summarizing a 6,000-word document instead of a tweet, the KV cache for that single batch ballooned, causing the container to run out of memory (OOM) mid-generation and taking down every other in-flight request in the same batch. This failure mode is more severe than a typical web service OOM because it not only drops the new request but also terminates queries that were already halfway through generating answers for paying customers. What We Actually Changed The readiness adjustment turned out to matter more than any Docker flag. We split liveness from readiness: liveness just checks that the process hasn't died; readiness fires a real, tiny generation request through the local API and only flips to healthy once that round trip succeeds. That alone killed the cold-start routing problem because the load balancer stopped trusting a merely alive process. We also gave up on bin-packing two replicas per GPU. In hindsight, treating GPU memory like it's as elastic as CPU or RAM was the actual root cause, not any single Docker setting. We implemented a model that uses one GPU, sets a conservative memory utilization ceiling, and enforces a request-level token limit at the proxy in front of the container, rather than inside it, because it is too late to make adjustments once the batch is already running. On the orchestration side, we stopped trying to scale-to-zero or scale aggressively off CPU-style metrics. Scale-to-zero is effective for web apps but doesn’t fit GPU-bound LLM serving, where cold starts can outlast traffic spikes. We kept a warm floor of replicas sized to baseline traffic and let a request queue absorb bursts instead of expecting new pods to materialize in time. It's less elegant than the autoscaling story everyone likes to tell, and it costs more in idle GPU time, but it's honest about what the hardware can actually do. What We Rejected, and Why We seriously considered dropping self-hosting altogether and routing through a managed inference API. For a side project, that's probably the right call — less to own, no GPU bin-packing headaches. We rejected it due to data residency requirements that prohibited sending raw text to a third party, and at our volume, managed pricing would quickly exceed our GPU costs. We also looked at Ray Serve and Triton early on, and they solve some of the issues more natively, but the team's Docker and Kubernetes muscle memory was strong enough that rebuilding on a new serving framework felt like trading one set of unknowns for another, at least for the first version. Key Takeaways Never bake multi-gigabyte model weights into the image — there's no caching benefit at that size, only slower pushes and bigger registry bills.Set shared memory explicitly; vLLM and similar multiprocess servers will fail under load with Docker's tiny default.Treat GPU memory utilization conservatively and avoid bin-packing replicas onto a single GPU unless you can guarantee a strict ceiling per container.Build a readiness assessment that performs a real generation, not just a process check; cold model loading will otherwise receive routed live traffic.Don't expect autoscaling to save you on cold-start timescales measured in minutes; a warm floor plus a queue is more honest than reactive scaling. Closing Thought None of these issues was really a Docker failure; the container did exactly what we told it to do. The failure was treating a multi-gigabyte, GPU-bound, slow-to-warm process like it was just another stateless web container that happens to need a GPU flag. I suspect that many teams will learn this lesson in the same way we did, during an incident on a Friday afternoon. Is it the right move to keep stretching Docker and Kubernetes to fit LLM serving, or is this the workload that finally pushes most teams toward purpose-built serving layers?

Generative artificial intelligence introduces unprecedented unpredictability into software development pipelines. Traditional software returns predictable outputs for exact inputs. Large language models generate varied responses for the exact same prompt. QA teams face a massive challenge scaling quality checks for these probabilistic systems. Manual validation falls short during fast deployment cycles. Implementing LLM testing in CI/CD has become mandatory for any modern engineering team. This article outlines the exact workflow teams use to validate AI applications inside continuous integration pipelines. The Shift to Probabilistic Evaluation Models Testing deterministic applications relies on exact assertions. You pass an input and assert a literal string match or a numerical value. Testing generative AI applications requires a completely different approach. QA engineers must evaluate semantic meaning, tone, factual accuracy, and safety constraints. A simple string-matching test fails when the model rewrites a correct answer using different synonyms. Establishing Semantic Metrics Teams need a comprehensive test automation built exclusively for natural language evaluations. Integrating these specialized frameworks directly into the deployment pipeline catches hallucinations early. Evaluating AI outputs involves a mix of quantitative metrics (like mathematical scores) and qualitative assessments (such as human review and LLM-based evaluation). QA teams calculate a similarity score between the generated text and a known good reference. A cosine similarity check determines how close the meanings are. Defining the Deployment Gate Exact word matching fails frequently with language models. The testing pipeline relies on semantic similarity to determine whether a build passes or fails. This mathematical approach significantly reduces human subjectivity from the testing process, instead of removing it completely. Engineering teams set hard thresholds for these similarity scores within the deployment configuration. Step-by-Step Workflow for LLM Testing in CI/CD Setting up LLM testing in CI/CD starts with defining clear evaluation metrics. Teams typically measure faithfulness, answer relevance, and context precision. Faithfulness checks if the model bases its answer strictly on the provided context. Answer relevance evaluates if the response directly addresses the user's prompt without going off-topic. Phase 1: Version Control and Prompt Linting The QA workflow begins the moment a developer commits code. Modern AI applications store prompts as version-controlled assets alongside application logic. The automated sequence triggers an initial static analysis phase upon the pull request. Linter tools check the prompt templates for missing variables or formatting errors. This step prevents broken prompts from reaching the model API. Minimizing Early Execution Costs Catching basic structural errors early saves significant API costs and execution time. Executing live model calls during every single unit test creates unacceptable delays and unpredictable overhead. A mature test automation uses mocked responses for basic functional validation. To ensure these mocks remain accurate, teams often implement schema and contract validation. The pipeline injects predefined JSON responses to verify the application's parsing logic while simultaneously validating them against a formal contract (e.g., OpenAPI or JSON Schema). Phase 2: Mocked Unit Tests QA engineers validate that the application handles API timeouts, rate limits, and malformed outputs correctly. This isolates the application code from the model's unpredictability. Fast feedback loops at this stage keep developers productive. Developers receive immediate alerts if their code breaks the fundamental integration points. Phase 3: Automated Evaluation Runs The core of LLM testing in CI/CD happens during the integration phase. The pipeline deploys the application code to an ephemeral staging environment. The testing script pulls a curated dataset of diverse prompts representing real-world user queries. The system sends these prompts to the live endpoint and records the generated responses. Evaluation frameworks score these responses against the predefined metrics. Pipeline Configuration Example Implementing this layer demands deep large language model optimization to balance test execution speed with evaluation accuracy. Let us look at a practical implementation snippet for GitHub Actions. This configuration installs the required Python dependencies first. The sequence then executes the automated AI evaluations and generates an XML report. YAML name: LLM Pipeline Evaluation on: [push] jobs: evaluate-llm: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Setup Python environment uses: actions/setup-python@v4 with: python-version: '3.10' cache: 'pip' # Enables dependency caching - name: Install evaluation dependencies run: pip install -r requirements-test.txt - name: Run automated AI evaluations env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY } # Added --instafail and --retries for flaky API stability run: | pytest tests/llm_evaluations/ \ --junitxml=reports/result.xml \ --force-sugar - name: Upload Evaluation Report if: always() # Ensures reports upload even if tests fail uses: actions/upload-artifact@v4 with: name: llm-eval-report path: reports/result.xml Phase 4: Guardrail and Toxicity Checks Security and safety form a mandatory layer of testing generative AI applications. The pipeline executes specialized tests to attempt prompt injection and jailbreak scenarios. The test suite verifies that the model rejects malicious requests and refuses to generate toxic content. PII scanners analyze the outputs to prevent data leaks. Any failure in this security phase triggers an immediate halt. Protecting Brand Integrity Securing the application boundary protects the brand reputation and user privacy. Malicious actors constantly look for vulnerabilities in AI endpoints. Automated security gating stops these vulnerabilities from reaching production. Phase 5: Regression Tracking and Deployment The pipeline aggregates the evaluation scores and compares them against previous test runs. A sudden drop in the faithfulness metric indicates a regression in the model prompt or the retrieval logic. QA teams use specialized dashboards to visualize these trends over time. Only when the new build meets the historical baseline does the process proceed. Closing the Feedback Loop The code is then merged and deployed to the production environment automatically. Continuous monitoring takes over to track live user interactions. Production telemetry feeds directly back into the QA workflow. Engineering teams extract challenging edge cases from live user logs. QA engineers format these edge cases into new test scenarios for the automated suite. Architecting the Testing Infrastructure This continuous feedback loop drives ongoing optimization of large language models. The test dataset grows more comprehensive with every deployment iteration. Refining the dataset guarantees the test automation framework remains highly relevant to actual user behavior. Provisioning Dedicated Hardware Running AI evaluations demands significant computational resources. Standard runners often lack the memory to process large datasets quickly. Engineering teams provision specialized runner instances equipped with high memory capacity. These dedicated machines execute the scripts much faster than standard nodes. Faster execution times prevent bottlenecks during the integration phase. Addressing Cost and API Latency Optimizing the hardware infrastructure directly supports seamless workflows. Teams parallelize test execution across multiple runners to reduce total build time. Executing hundreds of live calls in a CI/CD pipeline costs money and takes time. Teams must optimize their testing strategies to manage these constraints. Tiered Execution Strategies Running a smaller smoke test suite during regular commits provides immediate feedback. The full regression suite runs overnight or before major releases. Caching previous model responses for identical prompts reduces unnecessary API calls. Engaging with QA teams for AI applications helps teams effectively architect tiered testing strategies. Resolving Non-Deterministic Test Failures Traditional automated tests pass or fail consistently. Generative models introduce flakiness into the pipeline by their very nature. A prompt might pass the similarity threshold on nine attempts and fail on the tenth. QA engineers implement retry logic explicitly for these edge cases. Smart Retry Mechanisms The test script requests a regeneration from the model up to three times before registering a hard failure. Analyzing the failure logs helps teams identify poorly phrased prompts. Continuous refinement of both the application code and the test scripts eliminates instability. Scaling the Workflow and Advanced Techniques Integrating LLM testing in CI/CD requires constant adaptation. New foundational models are released frequently. API endpoints change, and token limits shift. The QA architecture must remain modular to enable seamless swapping out of underlying models. Isolating the evaluation logic from the application code prevents vendor lock-in. Shadow Deployments A well-architected pipeline treats the AI model as just another replaceable microservice. Advanced teams implement shadow deployments for major model upgrades. The pipeline routes a small percentage of live traffic to the new model version alongside the existing one. Automated scripts compare the responses from both versions in real-time. This provides statistical confidence before fully committing to the update. Managing Hallucinations at Scale Testing generative AI applications in production via shadow traffic uncovers nuances missed in staging environments. Monitoring actual user interactions reveals unexpected model behaviors. Hallucinations remain the primary risk factor for AI deployments. The automated sequence must include dedicated datasets designed exclusively to trigger known hallucination patterns. Optimizing the RAG Pipeline Modern AI relies on retrieval-augmented generation (RAG) for accurate context, meaning the language model isn't the only component needing testing. The CI/CD workflow must evaluate the retrieval database independently to ensure performance. QA teams use tests to verify that vector databases return the correct document chunks for specific queries. By measuring metrics like Precision (relevance) and Recall (comprehensiveness), teams can fine-tune embeddings and retrieval parameters to ensure the most accurate data reaches the model. Conclusion Integrating LLM testing in CI/CD transforms unpredictable AI experiments into dependable enterprise software. Implementing structured metrics, mocked unit tests, and automated evaluation scripts secures the deployment pipeline. Teams that adopt specialized test automation ship high-quality AI features with complete confidence. Automating the evaluation of semantic accuracy and security creates a predictable release cadence. Engineering organizations must embrace these testing paradigms to lead the generative AI market. Adopting this structured workflow guarantees consistent product quality across every release.