Every few years, a mature technology finds an unexpected use that breathes new life into the data profession. Change data capture did it for data delivery, and cloud data warehouses did it for business analytics. You may think I’m going to wax lyrical about generative AI, but I’m actually thinking of a markup language that's “having a moment” thanks to generative AI. What follows is an appreciation of the YAML format and how it’s quietly revolutionizing agentic data engineering.
It all begins with YAML
First proposed in 2001 by Clark Evans, Ingy döt Net, and Oren Ben-Kiki, YAML was originally short for "Yet Another Markup Language" before being recast as the recursive "YAML Ain't a Markup Language" to stress that it is built for data uses rather than documents. Today, YAML is the de facto configuration language that defines everything from Kubernetes manifests and Docker Compose files to CI/CD workflows and Ansible playbooks. Its popularity comes down to a few practical benefits: it’s human-readable, simple enough to learn in an afternoon, supports schema validation, is easy for machines to parse, and lives in plain-text files that fit naturally into existing version-control workflows. Moreover, while those benefits have driven adoption by software engineers, they apply equally, if not more so, to agentic technologies, with YAML finding product/market fit for many AI uses.
AI and data engineering redux
If you missed Dan Potter's latest blog post, “Agentic AI Just Rewrote the Data Engineer’s Job Description,” I urge you to read it to understand why Agentic AI is profoundly reshaping the traditional data engineering role. The great news is that Dan argues AI automation won’t make data engineers obsolete. Instead, they will become more valuable by using Agentic AI to amplify their skilled judgment. Dan also highlights three tectonic shifts driving this revolution: intent-driven design, autonomous operations, and a trusted data foundation. Together, these concepts let data engineers tell platforms what the source data is, how it’s merged, and what the curated result should look like. So, data engineers describe the outcome and let the underlying platform technology, more specifically the data pipeline agents, figure out how.
The multiplier, not the shortcut
Not surprisingly, coding agents consume and produce YAML files to enable intent-driven development. The data engineer defines the pipeline goals, the constraints, and the quality rules in natural language. The coding agent then generates declarative, YAML-based pipelines that are readable enough for AI to build, robust enough for runtime processes to interpret, and clear enough for people to govern.
Write a pipeline as an intent rather than a set of instructions, and you get something that compounds. The same definition tells the engine how to refresh incrementally, the orchestrator the dependencies, the reviewer what changed, the auditor what the data means and where it came from, and a coding agent exactly what it can generate next. One artifact, five audiences. In traditional programming (an imperative world), each of those was a separate document, a separate script, or a person's memory.
The compounding occurs in pipeline runtime engines, too. A declarative definition survives platform changes, schema drift, and an engineer's departure far better than a procedure does, because it records the business outcome rather than the mechanics someone chose one afternoon in 2015. And because the definition is readable by a machine, every improvement to the engine or the agent underneath it lifts every pipeline you have already written. You essentially get better at data engineering while you sleep!
For agentic AI specifically, this is the whole game. Coding agents are superb at producing declarative artifacts and unreliable at reasoning about imperative side effects. Give them a schema-validated pipeline language, and they become tireless development team members. Declarative pipelines are how you turn one engineer plus one coding agent into a productivity machine.
Vendor and platforms with various routes up the same mountain
Good news for anyone planning a modern estate: several major data platforms and open-source projects have converged on this direction in 2026, each from a different starting point.
Apache Camel, the open-source integration framework, is a clear example of YAML in action and has been an early adopter. Since Camel 3.9, in early 2021, its YAML DSL has let developers define routes, route templates, Kamelets, and REST configuration as plain YAML files instead of ‘classic’ Java or XML. While there are no authoritative statistics on how many Camel developers use YAML versus Java DSL, the project’s current trend and tooling strongly favor YAML for many workflows in the agentic era.
Databricks has pushed hardest on execution semantics. Lakeflow Spark Declarative Pipelines let you define streaming tables and materialized views in SQL or Python while the runtime handles incremental processing, some change data capture, and dependency ordering. Databricks contributed the authoring framework to Apache Spark as Spark Declarative Pipelines.
Snowflake acquired TensorStax in February to layer an agent on top of its declarative objects to write, test, and repair pipelines across various tools. However, YAML plays a supporting role as a configuration mechanism rather than serving as the pipeline language itself. The TensorStax tooling now lives inside Cortex Code (CoCo), where it sits alongside Snowflake's declarative building blocks: Dynamic Tables defined in SQL, dbt projects configured in YAML, and YAML specifications for semantic views and agents.
Qlik’s unique position
Furthermore, future data estates won't be single-platform, and they won't become single-platform. Databricks and Snowflake sit side by side, an Iceberg lakehouse is often being stood up next to them, and the data feeding all three lives in SAP, Oracle, mainframes, and a few hundred SaaS applications that none of those platforms can see natively.
Each platform has made transformation declarative inside its own boundary and done it well. The multiplier grows when the pipelines that cross those boundaries are declarative, too.
That is the job Qlik Talend Cloud does: log-based change data capture from the systems of record, streaming into Qlik Open Lakehouse or directly into Databricks and Snowflake, with the same YAML definitions, the same GitHub workflow, and the same governed context applied from the first byte to the last table. A Databricks materialized view and a Snowflake Dynamic Table are only as fresh and as trustworthy as the bronze layer beneath them. Make that layer declarative and agent-legible, and everything built on top inherits the benefit.
Qlik has approached declarative pipelines as an independent vendor that does not own your warehouse or Lakehouse, a position that has proven advantageous. At Qlik Connect in April, we introduced declarative pipelines defined in YAML as a natural-language-driven approach to creating and evolving Qlik data pipelines that support both the Databricks and Snowflake platforms, among many others.
On June 30, our declarative pipeline feature became generally available, with the entire Qlik pipeline project expressed as YAML and published schemas that provide autocomplete and live validation in your preferred development environment, such as VS Code. We support the most popular coding agents, including Claude Code, Codex, and GitHub Copilot, and we’ve recently added new data pipeline tools to the Qlik MCP server so agents can retrieve context, including pipeline projects, data connectivity, lineage, and data quality scores, to improve reasoning and YAML code generation.
This is what complementary looks like in practice. Not simply another place to do transformation, but the piece that makes declarative pipelines an estate-wide property instead of a platform feature.
Where does this go next?
I am optimistic about what declarative YAML and coding agents do for the profession. The imperative era rewarded engineers for knowing how. The declarative era hands the how to the platform and the agent, and hands the engineer something better: deciding what the data should mean, how fresh it must be, which quality rules exist, and where a human must approve before an agent acts.
I expect the next eighteen months to move quickly. Coding agents will graduate from generating pipelines to proposing them, from proposing to maintaining, and every pipeline declared today is one an agent can improve later.
Try it today
Qlik Talend Cloud declarative pipelines are available for you to try today. Start a free trial here, load the Qlik Pipeline YAML into your favorite ide, connect the Qlik MCP server to your preferred coding agent, and use GitHub for code collaboration. Then just use natural language prompts to build your data pipelines!

