Data Engineer / Analyst
AI drafts the SQL; you define meaning and check the numbers
Print / Save as PDF
Top 3 changes for you
- 1Writing SQL, transformation models and pipeline codeDaily AI use is common; organisational adoption lags
- 2Answering business questionsGenerated SQL can run fine yet return wrong results
- 3Defining metrics and business meaningBetter data modeling can help AI answer more questions
- Tools to try first
- GitHub Copilot(opens in new tab)dbt(opens in new tab)
- Skills to build
- Writing semantic models and business definitions that AI tools can use
- Checking AI-generated SQL for wrong joins and misread columns
How your work changes
#Open any item to see before and now, with sources.
BuildWriting SQL, transformation models and pipeline codeMost surveyed practitioners now use AI tools daily; organisations lag behind
Before
Data engineers, analytics engineers and analysts wrote SQL, transformation models, pipeline code, tests and documentation by hand, and teammates reviewed the changes.
Now
In Joe Reis's 2026 State of Data Engineering Survey (about 1,100 responses; the post gives both 1,101 and 1,001), 82% use AI tools daily or more and only 3.7% find them unhelpful. Organisational adoption lags behind, with 64% still experimenting or using AI for tactical tasks only. In dbt Labs' 2026 State of Analytics Engineering survey of 363 data practitioners and leaders, 72% prioritise AI-assisted coding within their development process.[1][2]
BuildBuilding ELT pipelines with AI agentsOn a benchmark, agents largely solve extract and load; transformation improves
Before
Engineers built Extract-Load-Transform pipelines by hand, which is labour-intensive work, setting up extraction and loading from source systems and then writing the transformation models.
Now
On ELT-Bench, the first benchmark for end-to-end ELT pipeline construction, AI agents initially showed low success rates. A 2026 re-evaluation with upgraded language models found that the extraction and loading stage is largely solved, while transformation performance improves significantly. The same study found that most failed transformation tasks contained benchmark errors, including rigid evaluation scripts, ambiguous specifications and incorrect ground truth, that penalised correct agent output.[6]
TestTesting and monitoring pipelinesIn dbt's survey, validation investment isn't keeping pace with AI output
Before
Teams added data tests and monitoring to pipelines and investigated failures when a check or a stakeholder flagged bad data.
Now
In dbt Labs' 2026 survey, only 24% prioritise AI-assisted pipeline management, including testing, observability and quality controls. The report says AI is scaling engineering throughput and stakeholder-facing outputs, but investment in validation, testing and governance mechanisms isn't scaling at the same rate. A significant majority of respondents are concerned about hallucinated or incorrect data reaching stakeholders.[1]
OperateAnswering business questionsAI-generated SQL might run fine yet return wrong results
Before
Business users sent questions to analysts, who found the right tables, wrote the SQL and returned a number, chart or dashboard.
Now
With text-to-SQL, a language model generates the SQL from the question. dbt Labs notes that the model might join tables incorrectly, misinterpret a column's meaning, or produce a query that runs successfully but returns wrong results. A 2026 research paper argues that NL2SQL agents still make mistakes on large-scale real-world databases because they lack knowledge of how to use the underlying data, such as the intent of each column, and form misconceptions about the data when querying it. In a small NYU study (a preprint; 20 SQL-literate professionals and graduate students, using a GPT-4o-backed tool in spring 2025), the natural-language interface reallocated the work of querying rather than removing it: users still had to verify the generated SQL.[3][4][7]
DesignDefining metrics and business meaningIn dbt's benchmark, added modeling helped both semantic layer and text-to-SQL
Before
Analytics engineers defined metrics and business logic in data models, BI tools and documentation so that reports used the same definitions.
Now
A 2026 research paper argues that NL2SQL agents make mistakes on large real-world databases partly because they lack knowledge of how to use the data, such as the intent of each column. In dbt Labs' own 2026 benchmark update (11 questions, each run 20 times), adding three models let its Semantic Layer cover every question on its single ACME Insurance dataset, and text-to-SQL improved too. Snowflake's documentation says schemas lack knowledge like business process definitions and metrics handling, and Cortex Analyst uses a semantic model to bridge that gap.[3][7][8]
TestChoosing and evaluating an AI query toolTwo text-to-SQL benchmarks had high annotation error rates that moved rankings
Before
Teams compared query and BI tools on features, performance and fit with their existing data stack.
Now
The research community depends on public text-to-SQL benchmarks and leaderboards to compare techniques and select one for deployment. A 2026 expert analysis found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. When the authors re-evaluated 16 open-source agents from the BIRD leaderboard on a corrected subset of the BIRD Dev set, relative performance changed by −7% to 31% and rankings moved by up to nine positions. Snowflake's documentation describes evaluating a semantic view by running your verified queries against Cortex Analyst.[5][8]
Risks to watch
#AI-written SQL can run without errors and still return the wrong number
AI-generated SQL can run without errors and still return the wrong number. dbt Labs notes that with text-to-SQL, failure looks like a plausible but incorrect answer, while with its Semantic Layer failure looks like an error message.[3]
A semantic layer only covers what has been modeled, dbt Labs' benchmark shows
A semantic layer only covers what has been modeled. In dbt Labs' benchmark, questions the Semantic Layer could not answer in 2023 still could not be answered without additional modeling. Its text-to-SQL runs loaded the entire schema as context, which dbt Labs says isn't practical for larger datasets.[3]
Leaderboard scores are a weak basis for choosing a tool, as benchmarks can contain errors
Leaderboard scores are a weak basis for choosing a tool. Researchers found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow, and warn that annotation errors can distort reported performance and rankings and could misguide deployment choices.[5]
Output can grow faster than checking, dbt Labs' survey suggests
Output can grow faster than checking. In dbt Labs' 2026 survey, 72% prioritise AI-assisted coding but only 24% prioritise AI-assisted pipeline management, and a significant majority are concerned about hallucinated or incorrect data reaching stakeholders.[1]
Sensitive data can end up in AI tools, a concern raised in dbt Labs' survey
Sensitive data can end up in AI tools. In dbt Labs' survey, practitioners showed a higher level of concern than leaders about exposing sensitive data to LLMs.[1]
Skills to build
#Human skills: what's new, what matters more, and what AI now does for you.
New skillsDidn't really exist in the role before
- Writing semantic models and business definitions that AI tools can use
- Checking AI-generated SQL for wrong joins and misread columns
- Building a set of verified questions and answers to evaluate query tools
- Reviewing agent-generated pipeline and transformation changes
- Deciding which questions AI may answer without an analyst
More important nowExisting skills that AI makes more valuable
- Data modeling
- Data testing and observability
- Metric definitions and data ownership
- Knowing the quirks of your own data
- Data governance and access control
- Explaining uncertainty in a number to stakeholders
Less centralStill useful, but AI now does much of it
- Writing boilerplate SQL and YAML by hand
- Hand-writing extraction and loading code for common sources
- Answering routine one-off data requests
Tools
#"Start here" tools are the editor's pick of where to begin for this role. Our suggestion
Start here
Widely used and core to this role
- GitHub Copilot(opens in new tab)
· AI coding assistant
Drafting SQL, Python and pipeline code in the IDE with completions, chat and agent mode, with changes reviewed in a pull request.
- dbt(opens in new tab)
· Data & analytics
Transformation models with tests, documentation and a Semantic Layer. dbt Copilot generates SQL, documentation, tests and semantic models, and the open-source dbt MCP server gives AI clients project metadata, Semantic Layer queries and dbt commands.
Also worth a look
Depends on your team and stack
Show 5 toolsHide tools
- Claude Code(opens in new tab)Anthropic
Agentic work in a data or dbt repository from the terminal or IDE, such as building, debugging and refactoring models and pipeline code.
- Cortex Analyst(opens in new tab)Snowflake
Answering business questions in natural language over Snowflake data using a semantic view, and evaluating it with verified queries.[8]
- Genie Agents(opens in new tab)Databricks
Natural-language questions over Databricks data, with example queries, text snippets and SQL functions added by subject matter experts, governed through Unity Catalog.
- Hex(opens in new tab)Hex Technologies
Notebooks with AI assistance for code-based analysis, Threads for business users to ask questions in plain language, and semantic models to govern AI answers.
- Monte Carlo(opens in new tab)Monte Carlo
Table monitors that learn normal update, size and growth patterns and alert on breaks, plus a Troubleshooting Agent (in preview) that traces an alert's root cause.
Your first steps
#- 1Check which AI tools you may use and which warehouse data, schemas and query results may be sent to them.
- 2Write 20 to 30 real business questions with verified answers and run any AI query tool against them before rolling it out.
Show 3 more stepsShow fewer steps
- 3Pick your five most-used metrics and make sure each has one written definition in your semantic layer or models, with an owner.
- 4Review the last few AI-assisted changes to your pipelines and add tests where a wrong join or duplicate rows would go unnoticed.
- 5Try an agent on one well-understood transformation task and compare its output row by row with the existing model.
A week in this role now
A typical week now involves less typing of SQL and more reading of it. An assistant drafts a new model or a fix to a failing pipeline, and you check the joins, the grain and the row counts before the change goes to review. When a stakeholder asks a question, they may have already asked an AI tool and come to you with a number. Part of your job is to find out where that number came from and whether it used the right definition.
More of the week goes into the things AI tools depend on: clear metric definitions, documented tables, tests that catch duplicate rows or late data, and a set of known questions with verified answers to try new tools against. Agents can take on routine pipeline work, but you still own what reaches a dashboard or a board deck. Treat a confident answer with no visible query or definition behind it as unchecked until you have checked it.
Agent Skills
AI instructions to copy
#Each one is a text file (SKILL.md) that tells an AI tool, step by step, how to do one task the way you want. Copy it into a chat, or add it to a tool that supports Agent Skills. Which tools, and how
Starter instructions for this role
Our suggestionWritten from the changes above. Read and adapt one before you use it.
SQL answer sanity checkChecks a SQL query and the number it returns before the answer goes to a stakeholder
How to use it
- In any AI chat, such as ChatGPT, Microsoft 365 Copilot or Claude: choose Copy instructions, paste them at the start of a new chat, then add your material. Our suggestion
- Install it in Claude (web or desktop): download the .zip, then go to Customize › Skills › + › Create skill › Upload a skill. Code execution must be turned on. Claude help (opens in new tab)
- Install it in a coding agent such as Claude Code, GitHub Copilot or Codex: unzip the folder into the tool's skills folder. Folder locations
Show the full file
File name
SKILL.md, in a folder namedsql-answer-sanity-check. When to use it: Checks a SQL query and the number it returns before the answer goes to a stakeholder, covering joins, grain, filters, metric definitions and simple reconciliation checks. Use when an AI tool or a colleague has written a query that answers a business question, when a stakeholder brings a number from an AI assistant, or when someone asks "is this number right?", "check this SQL", "validate this query" or "sanity check these results".--- name: sql-answer-sanity-check description: Checks a SQL query and the number it returns before the answer goes to a stakeholder, covering joins, grain, filters, metric definitions and simple reconciliation checks. Use when an AI tool or a colleague has written a query that answers a business question, when a stakeholder brings a number from an AI assistant, or when someone asks "is this number right?", "check this SQL", "validate this query" or "sanity check these results". metadata: title: "SQL answer sanity check" summary: "Checks a SQL query and the number it returns before the answer goes to a stakeholder" --- # SQL Answer Sanity Check Confirm that a query answers the question that was actually asked, and that its result is plausible, before anyone relies on it. ## What to gather first Ask for anything missing before you start: - The business question in the asker's own words, and who will use the answer. - The SQL query, and the result it returned (or a sample of rows). - The agreed definition of any metric involved (for example "active customer" or "net revenue"), and where that definition lives. - The tables used, with their grain (what one row represents) and primary keys, if documented. - A trusted reference number to compare against, such as an existing dashboard, report or last period's figure. - Whether you may run read-only queries, and which data you are allowed to see. Ask for anything missing once. If the user can't provide it or asks you to go ahead, proceed and record every assumption. ## Steps 1. Restate the question as a precise spec: the metric, the time window, the time zone, the filters, the grouping and the unit. Note any word in the question that could mean more than one thing. 2. Read the query clause by clause and write down, in plain English, what it actually computes. Compare this with the spec from step 1. 3. Check the grain of every table in the query. For each join, state the join keys and whether the relationship is one-to-one, one-to-many or many-to-many. 4. For any join that could multiply rows, draft a read-only check: count rows before and after the join, and count distinct keys against total rows. 5. Check join types. Flag inner joins that silently drop rows with no match, and left joins whose later `WHERE` clause turns them back into inner joins. 6. Check filters: date boundaries (inclusive or exclusive), time zones, soft-deleted or test records, status values, and how NULLs are treated in filters and aggregates. 7. Check the metric logic against the agreed definition: `COUNT` versus `COUNT(DISTINCT)`, sums of values that are already aggregated, averages of averages, currency and unit conversions. 8. Draft reconciliation checks: total the result and compare it with the reference number; check that segment subtotals add up to the total; spot-check a few individual records by hand. 9. If you are allowed to, run only the read-only checks (`SELECT` statements) on a limited sample. Use a development copy or replica where possible, add LIMIT or sampling, and ask before scanning large tables. Never run statements that write, update, delete or change permissions. 10. Decide a verdict: looks right, needs a fix, or cannot confirm. If a fix is needed, propose a corrected query and explain each change. ## Output format ```markdown ## Sanity check: <short name of the question> **Question as understood:** <one sentence spec> **Verdict:** Looks right | Needs a fix | Cannot confirm ### What the query actually computes <plain-English description> ### Checks | Check | What I looked at | Result | Notes | | --------------------------- | ---------------- | ---------------------------- | ----- | | Grain and joins | | Pass / Concern / Not checked | | | Row multiplication | | | | | Filters and dates | | | | | NULL handling | | | | | Metric definition | | | | | Reconciliation to reference | | | | ### Issues found 1. <issue> — <why it matters> — <suggested fix> ### Suggested query (if changed) <SQL with comments on each change> ### Assumptions and open questions - <assumption or question, and who can confirm> ``` ## Watch out for - A query that runs without errors can still return the wrong number. A plausible result is not evidence that it is correct. - Column names can mislead. A column called `status` or `amount` may not mean what its name suggests; check documentation or ask the owner rather than guessing. - Fan-out from one-to-many joins inflates sums and counts without any visible error. - Different teams may use different definitions of the same metric. Use the agreed definition, and say so if none exists. - Late-arriving or incomplete data can make the most recent period look low. - Never reproduce secrets, credentials or personal data in the output unless the user needs them. Summarise, mask identifiers, or use counts. - If the question cannot be answered from the modeled data, say so instead of stretching the query to fit. ## Before you finish - [ ] The question spec and the query's actual behaviour are both written down and compared. - [ ] Every join has a stated grain and a row-multiplication check. - [ ] The result was reconciled against a reference, or the output says no reference was available. - [ ] No write, update or delete statements were run or suggested for running. - [ ] No sensitive data appears in the output. - [ ] Clearly flag every assumption, and anything a human must review or decide, such as an ambiguous definition or a number that could not be confirmed.
Pipeline change reviewReviews a data pipeline change for errors, downstream impact and missing tests
How to use it
- In any AI chat, such as ChatGPT, Microsoft 365 Copilot or Claude: choose Copy instructions, paste them at the start of a new chat, then add your material. Our suggestion
- Install it in Claude (web or desktop): download the .zip, then go to Customize › Skills › + › Create skill › Upload a skill. Code execution must be turned on. Claude help (opens in new tab)
- Install it in a coding agent such as Claude Code, GitHub Copilot or Codex: unzip the folder into the tool's skills folder. Folder locations
Show the full file
File name
SKILL.md, in a folder namedpipeline-change-review. When to use it: Reviews a proposed change to a data pipeline or transformation model, often drafted by an AI agent, for correctness, grain changes, downstream impact and missing tests, and produces a review with suggested tests. Use when reviewing a pull request that touches SQL models, ELT jobs, schema changes or pipeline code, or when asked to "review this model change", "check this pipeline PR" or "what could this change break downstream".--- name: pipeline-change-review description: Reviews a proposed change to a data pipeline or transformation model, often drafted by an AI agent, for correctness, grain changes, downstream impact and missing tests, and produces a review with suggested tests. Use when reviewing a pull request that touches SQL models, ELT jobs, schema changes or pipeline code, or when asked to "review this model change", "check this pipeline PR" or "what could this change break downstream". metadata: title: "Pipeline change review" summary: "Reviews a data pipeline change for errors, downstream impact and missing tests" --- # Pipeline Change Review Review a change to a pipeline or transformation model so that wrong joins, duplicate rows or broken downstream reports are caught before the change is merged. ## What to gather first Ask for anything missing before you start: - The diff or pull request, and the stated reason for the change. - Whether the change was written by a person, an AI agent or both. - The current version of each changed model or job, and its documented grain and keys. - The list of downstream models, dashboards or exports that depend on the changed outputs, if available. - Existing tests for the affected models. - Whether you may run read-only comparison queries in a development or staging environment. Ask for anything missing once. If the user can't provide it or asks you to go ahead, proceed and record every assumption. ## Steps 1. Summarise the intent of the change in two or three sentences, and check that the diff does what the description says and nothing more. 2. List every changed output: tables, columns, data types, and column meanings. Mark each as added, removed, renamed or changed. 3. For each changed model, state the grain before and after the change. Treat any grain change as a high-risk finding. 4. Review joins, filters, deduplication logic and incremental logic (for example, how late or updated records are handled). Note any place where rows could be duplicated or dropped. 5. Trace downstream impact: which dependents read the changed columns, and what they would see after the change. 6. Compare with the existing model where possible. Draft read-only queries that compare row counts, distinct key counts, and totals of key measures between the old and new versions. 7. Check tests. List which existing tests cover the change, and propose missing ones: uniqueness and not-null on keys, accepted values, relationships between tables, and row count or freshness expectations. 8. Check for secrets, credentials, hard-coded environment names, or personal data newly exposed in outputs. 9. Check whether documentation and metric definitions need an update to match the change. 10. Write the review. Do not merge, deploy, backfill or drop anything; those are decisions for the owner. ## Output format ```markdown ## Pipeline change review: <PR or change name> **Intent:** <2–3 sentences> **Recommendation:** Ready to merge | Merge after fixes | Needs discussion ### Changed outputs | Object | Change | Grain before | Grain after | Downstream users | | ------ | ------ | ------------ | ----------- | ---------------- | ### Findings | # | Severity (High/Medium/Low) | Location | Finding | Suggested fix | | --- | -------------------------- | -------- | ------- | ------------- | ### Comparison checks (read-only) <queries to run, and results if they were run> ### Missing tests to add - <test> on <model.column> — <what it would catch> ### Docs and definitions to update - <item> ### Assumptions and open questions - <assumption or question, and who can confirm> ``` ## Watch out for - Agent-generated changes can be larger than the task needed. Flag unrelated edits and ask for them to be split out. - Code that runs and passes existing tests can still be wrong if the tests never covered the changed logic. - A renamed or retyped column can break dashboards and exports that are not in the project's dependency graph. - Incremental models may look correct on a fresh build and still be wrong on the next incremental run. - Backfills and full refreshes can be expensive or irreversible. Recommend them only as a decision for the owner. - Never reproduce secrets, credentials or personal data in the output unless the user needs them. Use counts and masked examples instead of real customer records. ## Before you finish - [ ] Grain before and after is stated for every changed model. - [ ] Downstream impact is listed, or the output says it could not be traced. - [ ] Each high-severity finding has a suggested fix or a question for the author. - [ ] Missing tests are proposed for the risky parts of the change. - [ ] Nothing was merged, deployed, backfilled or deleted. - [ ] Clearly flag every assumption, and anything a human must review or decide before merging.
Metric definition write-upWrites one clear definition of a metric so people and AI query tools use the same meaning
How to use it
- In any AI chat, such as ChatGPT, Microsoft 365 Copilot or Claude: choose Copy instructions, paste them at the start of a new chat, then add your material. Our suggestion
- Install it in Claude (web or desktop): download the .zip, then go to Customize › Skills › + › Create skill › Upload a skill. Code execution must be turned on. Claude help (opens in new tab)
- Install it in a coding agent such as Claude Code, GitHub Copilot or Codex: unzip the folder into the tool's skills folder. Folder locations
Show the full file
File name
SKILL.md, in a folder namedmetric-definition-writeup. When to use it: Drafts a clear, unambiguous business definition for a metric, covering calculation, grain, filters, edge cases, owner and verified example questions, so people and AI query tools use the same meaning. Use when a metric is defined differently in different reports, before adding a metric to a semantic layer or data model, or when asked to "define this metric", "document what active users means" or "write a metric spec".--- name: metric-definition-writeup description: Drafts a clear, unambiguous business definition for a metric, covering calculation, grain, filters, edge cases, owner and verified example questions, so people and AI query tools use the same meaning. Use when a metric is defined differently in different reports, before adding a metric to a semantic layer or data model, or when asked to "define this metric", "document what active users means" or "write a metric spec". metadata: title: "Metric definition write-up" summary: "Writes one clear definition of a metric so people and AI query tools use the same meaning" --- # Metric Definition Write-up Turn a loosely named metric into a written definition that people and AI tools can apply the same way every time. ## What to gather first Ask for anything missing before you start: - The metric name and the business decision it supports. - Every existing place the metric is calculated: queries, models, dashboards, spreadsheets. - The source tables and columns involved, with their grain. - The person or team who owns the metric and can approve the definition. - Known disagreements or edge cases people have raised. - Where the final definition will live (your semantic layer, model documentation or a data catalog). Ask for anything missing once. If the user can't provide it or asks you to go ahead, proceed and record every assumption. ## Steps 1. Collect every existing calculation of the metric and put them side by side. 2. For each version, write down in plain English what it counts or sums, over which records, with which filters and time logic. 3. List every difference between the versions, such as included statuses, time zone, handling of refunds, test accounts or duplicates. 4. For each difference, list the options with trade-offs, and optionally mark a suggested one; the metric owner decides. Do not pick silently. 5. Draft the definition: a one-sentence meaning, the exact calculation, the grain, the time dimension, allowed breakdowns, and explicit inclusions and exclusions. 6. Write down edge cases and how the definition treats them (NULLs, cancelled or reversed records, late data, currency or unit changes). 7. Write three to five example business questions this metric should answer, with the expected answer shape, and mark which ones have a verified number. 8. Write questions this metric should not be used to answer, and point to a better metric if one exists. 9. If you are allowed to, draft read-only queries that show how far the existing versions differ on a recent period. 10. Produce the write-up as a draft for the owner to approve. Do not change live models or dashboards. ## Output format ```markdown ## Metric: <name> **Status:** Draft, awaiting approval by <owner> **Meaning:** <one sentence a business user would understand> ### Calculation - Measure: <what is counted or summed> - Grain: <one row = …> - Time dimension and time zone: <…> - Filters: <included and excluded records> - Allowed breakdowns: <dimensions> ### Edge cases | Case | Treatment | | ---- | --------- | ### Existing versions compared | Location | How it differs | Related decision | | -------- | -------------- | ---------------- | ### Decisions needed from the owner 1. <decision> - Option A: <…> — trade-off: <…> - Option B: <…> — trade-off: <…> - Suggested (optional): <option and one-line reason, or "None"> - Owner's decision: <to be decided by <owner>> ### Example questions | Question | Answer shape | Verified answer? | | -------- | ------------ | ---------------- | ### Do not use this metric for - <question> — use <other metric> instead ### Assumptions and open questions - <assumption or question, and who can confirm> ``` ## Watch out for - Schemas do not carry business meaning. Do not infer a definition from column names alone. - A definition only helps AI tools if it is written where they can read it and covers the questions people actually ask. - Two dashboards agreeing does not prove they are right; they may share the same mistake. - Changing a well-known metric can break trends people rely on. Note whether historical figures will shift. - Never reproduce secrets, credentials or personal data in the output unless the user needs them. Keep example answers free of sensitive data. ## Before you finish - [ ] Every existing version is listed and compared. - [ ] The calculation states grain, time logic, filters and edge cases explicitly. - [ ] Every open choice lists its options and trade-offs as a decision for the owner, not resolved silently. - [ ] Example questions are marked verified or unverified. - [ ] No live model, dashboard or definition was changed. - [ ] Clearly flag every assumption, and anything a human must review or decide before the definition is published.
Published skills you can reuse(2)From public skill catalogs. Check the license before use.
- jupyter-notebook(opens in new tab)openai/skillsOpen source
Creating and editing Jupyter notebooks for explorations and experiments.
- xlsx(opens in new tab)anthropics/skillsSource-available
Opening, cleaning and creating spreadsheets and CSV files.
Related roles
Related roles
Sources (8)
4 research papers, 2 surveys, 2 vendor pages · 4 vendor-affiliated
Sources (8)
4 research papers, 2 surveys, 2 vendor pages · 4 vendor-affiliated
Each source is labelled by what the page is and whether its publisher sells in this space. How we label sources
- 1.2026 State of Analytics Engineering Report(opens in new tab)
dbt Labs · 10 April 2026
SurveyVendor-affiliated
“While 72% prioritize AI-assisted coding, only 24% prioritize AI-assisted pipeline management, including testing, observability, and quality controls.”
- 2.The 2026 State of Data Engineering Survey (Interactive)(opens in new tab)
Practical Data (Joe Reis) · 10 February 2026
SurveyVendor-affiliated
“82% of you use AI tools daily or more. Only 3.7% find them unhelpful. But organizational adoption lags way behind. 64% are still experimenting or using AI for tactical tasks only.”
- 3.Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update(opens in new tab)
dbt Labs (Jason Ganz and Benoit Perigaud) · 7 April 2026
Vendor pageVendor-affiliated
“With text-to-SQL, failure looks like a plausible but incorrect answer.”
- 4.Natural Language Interfaces for Databases: What Changes for SQL-Literate Users?(opens in new tab)
arXiv (Panos Ipeirotis and Haotian Zheng, New York University) · 26 July 2026
Research paper
“The interface reallocated the work of querying rather than removing it: users still had to verify the generated SQL, and an NLIDB that hid it would remove the step that let them trust the answer.”
- 5.Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards(opens in new tab)
arXiv (Tengjun Jin, Yoojin Choi, Yuxuan Zhu and Daniel Kang) · 19 January 2026
Research paper
“Through expert analysis, we show that BIRD Mini-Dev and Spider 2.0-Snow have error rates of 52.8% and 62.8%, respectively. We re-evaluate all 16 open-source agents from the BIRD leaderboard on both the original and the corrected BIRD Dev subsets. We show that performance changes range from -7% to 31% (in relative terms) and rank changes range from -9 to +9 positions.”
- 6.ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities(opens in new tab)
arXiv (Christopher Zanoli, Andrea Giovannini, Tengjun Jin, Ana Klimovic and Yotam Perlitz) · 2 April 2026
Research paper
“First, re-evaluating ELT-Bench with upgraded large language models reveals that the extraction and loading stage is largely solved, while transformation performance improves significantly.”
- 7.Arming Data Agents with Tribal Knowledge(opens in new tab)
arXiv (Shubham Agarwal et al.) · 17 February 2026
Research paper
“Nonetheless, NL2SQL agents still make mistakes when faced with large-scale real-world databases because they lack knowledge of how to correctly leverage the underlying data (e.g., knowledge about the intent of each column) and form misconceptions about the data when querying it, leading to errors.”
- 8.Cortex Analyst(opens in new tab)
Snowflake Documentation
Vendor pageVendor-affiliated
“Generic AI solutions often struggle with text-to-SQL conversions when given only a database schema, as schemas lack critical knowledge like business process definitions and metrics handling.”