I have been dreaming about this project for years. The actual trigger for finally starting it is AI. Without it, I could not take on a project of this size alone.

I want to be transparent about that. Working with Claude and OpenAI Codex is what makes it realistic for me to attempt this. The idea has grown from one deployment script into a much bigger project covering several data platforms, deployment engines, assessment, testing, security, cost control, documentation, and catalogs for software and sample data. Codex also helped me research and edit this post.

Even this post took me several hours. AI wrote most of it, but I reviewed every part, added sources, and made my own edits until it said what I actually wanted to say.

That does not mean I am going to take unreviewed AI output and call it a release. I expect to spend months testing deployments, reviewing code, checking the architecture, and finding the places where an idea looked good on paper but did not survive contact with Azure.

What began as Azure SQLVM Toolkit has grown into a broader idea: Azure Data Lab Toolkit. Azure SQLVM Toolkit is not working yet. Before completing it, I decided to rebuild the architecture from the ground up so it can support the broader scope and hold up as I add more ideas later. The original project remains focused on SQL Server on an Azure virtual machine.

The larger goal is to build a practical, repeatable, secure, and cost-aware way to create data labs on Azure. I want to describe a lab, see what it might cost, understand the security and licensing implications, deploy it, prove that it works, and remove it cleanly afterwards. I also want to run the same experiment against two platforms and compare the results without rebuilding everything around the database by hand.

What I want the toolkit to cover

I want Azure Data Lab Toolkit to support labs built around:

The first target is still SQL Server on an Azure VM. It gives me a useful base for testing deployment, networking, Azure Bastion, Azure Key Vault, backup restore, guest setup, tooling, teardown, and CI/CD patterns. There is also a very practical reason: I work on a Mac, and a Windows lab gives me access to SQL Server Management Studio when I need it.

The new project is still at the beginning. Right now it contains an architecture scaffold, documentation, and a backlog, but no deployable cmdlets. It is not production ready. Everything below describes where I want to take it, not functionality that has already shipped.

The Azure SQLVM Toolkit documentation will remain online while I build the new project. The Azure Data Lab Toolkit documentation already explains the direction in more detail and includes a short pitch deck.

Data Labs need Sample Data

A lab without data is usually not much of a lab. I want a catalog that can bring in Microsoft samples, community databases, private backups, or files supplied by the user.

Community samples and tools will not be mandatory. Planned flags such as -UseCustomArtifact, -CustomArtifactPath, and -CustomArtifactUrl will let people bring their own sample data or software into a lab. Companies and developers will be able to use private backups, datasets, installers, scripts, or internal tools instead of choosing an item from the community catalog.

For many testing scenarios, community projects are still a great option. A well-known database or tool can provide a shared baseline for comparisons, and it can also become a useful starting point when developing an in-house solution.

Microsoft's SQL Server samples repository is an obvious starting point. That includes AdventureWorks, WideWorldImporters, and the older Northwind and pubs examples. Microsoft's own WideWorldImporters README credits Greg Low, Denzil Ribeiro, Jos de Bruijn, and Robert Cain as its authors. I want those names to stay visible instead of disappearing behind a download link.

The community has many excellent sources as well. Daniel Hutmacher publishes databases through SQL Sunday downloads, including a Chicago parking tickets database assembled from several public data sources and a meteorological database based on data from SMHI.

The SQLBI Contoso Data Generator comes from the SQLBI team founded by Alberto Ferrari and Marco Russo. Brent Ozar provides a useful SQL Server package of the Stack Overflow database, while the underlying content still belongs to its original authors and remains subject to the Stack Overflow content license.

PostgreSQL has a different community landscape. Its sample database index is a useful place to start, but projects such as Pagila, Chinook, and Northwind for PostgreSQL are community projects, not official PostgreSQL databases. Credit belongs with their maintainers: Devrim Gündüz and the Pagila contributors, Luis Rocha and the Chinook contributors, and Pascal Thomet and the Northwind adaptation contributors.

Microsoft also publishes Fabric samples, which can become useful starting points for guided Fabric labs rather than anonymous downloads hidden behind the toolkit.

Being listed here does not mean a database can automatically be redistributed. Every candidate still needs a review for licensing, attribution, integrity, size, compatibility, and security before the toolkit automates it.

Review plan before execution

The main input will be a schema-validated YAML file that can be read before it is run and kept in source control. Templates will provide easy starting points, and an optional terminal interface enabled with a flag will help people generate the YAML without forcing everyone through a wizard. Experienced users will still be able to use PowerShell flags for direct control.

The planned -Plan and -WhatIf flags will answer different questions. -Plan will work without Azure access. It will validate the configuration, resolve defaults and derived values, and explain how a small set of base variables becomes the resource names, SKUs, network choices, catalog selections, guardrails, and deployment actions in the final plan.

-WhatIf will be Azure-aware. After the user signs in, it will compare the resolved plan with the resources that already exist and report what the toolkit would create, reuse, update, or flag as drift. It will not execute those changes. Together, the two flags will make both the toolkit's reasoning and the practical Azure impact reviewable before execution.

Before the toolkit deploys anything, it will create a plan showing:

  • which resources will be created and which existing resources will be reused
  • what security, identity, networking, and secret-handling choices are being made
  • what the estimated cost is, including any budget, shutdown schedule, or time-to-live guardrail
  • which sample databases, tools, and private artifacts will be installed and where they come from
  • which licenses, third-party terms, integrity checks, or sensitive-data warnings need attention
  • exactly what teardown will remove later

Cost estimation will be an explicit operation with its own flag, and HTML will be available as an output when the estimate or deployment report needs to be shared. The same run will also be able to produce useful console, JSON, and Markdown output.

Secrets will go into Azure Key Vault, with managed identities, private networking, and Azure Bastion used where they fit. More permissive choices, including printing a password to the shell, will require an explicit opt-in flag.

Only after the plan has been reviewed will deployment begin. Afterwards, probes will test the database, storage, and installed software. Teardown will start by listing what will be deleted and asking for confirmation.

PowerShell stays, but Bicep and Terraform will follow

The architecture has three parts: Core, Providers, and Engines.

The Core is the reusable part. It reads the YAML, resolves defaults, works with catalogs, creates the plan, handles assessment, cost and lifecycle rules, records run state, and coordinates teardown.

A Provider knows the details of one target. The SQL VM provider needs to understand images, storage, networking, SQL configuration, backup restore, and guest software. A Fabric provider has a different job again: workspaces, capacities, items, shortcuts, and deployment pipelines. A future Cosmos DB provider would eventually need to understand APIs, throughput, partitioning, and consistency choices. Those differences belong in providers instead of a growing collection of special cases in one script.

An Engine performs the approved actions. PowerShell comes first because I still want people to be able to use a PowerShell command and a YAML file without learning another infrastructure language before building a lab. I am also much better versed in PowerShell, and I think it still has a place here.

Bicep and Terraform will follow as additional deployment engines or export targets once the Core and provider contracts have proved themselves. The engine will be an explicit flag, not something the toolkit guesses. For teams operating a larger estate, one of those infrastructure-as-code routes will probably be a better fit. PowerShell, Bicep, and Terraform will still work from the same resolved plan instead of quietly producing three different labs.

Storage and software catalogs also need provider contracts.

CI/CD is part of the lab

A Git repository alone is not a CI/CD solution. The toolkit also needs repeatable workflows for reviewing changes, testing the module, checking security, approving deployments, collecting evidence, and proving that teardown worked.

GitHub will be the first Git provider, with GitHub Actions as the first pipeline engine. Pull requests will validate the YAML schema, run PowerShell and Pester checks, build the documentation, and perform dependency and vulnerability checks. That includes Dependabot for dependency maintenance and alerts, CodeQL where its supported languages apply, and Checkov for infrastructure-as-code scanning once the Bicep and Terraform engines exist.

Deployment workflows will use short-lived OpenID Connect (OIDC) federation instead of stored cloud credentials, require approval for sensitive environments, retain the resolved plan and reports as artifacts, run post-deployment probes, and record cleanup evidence. A failed deployment will leave enough state behind for investigation and a safe retry rather than simply stopping halfway through.

Azure DevOps with Azure Pipelines is the main Microsoft alternative. Later providers can cover GitLab CI/CD, Gitea Actions, and Forgejo Actions. Gitea and Forgejo are particularly interesting for self-hosted labs, where the Git service itself could run in Azure Container Apps.

Those systems use different workflow syntax, runners, secrets, and approval models. The toolkit will expose the same intent across them: validate, plan, approve, deploy, probe, report, and tear down. Fabric CI/CD is also part of the plan, but I do not yet know whether a complete Fabric CI/CD path can be implemented through infrastructure as code. I want to explore that in the future instead of presenting it as a solved part of the architecture. Whatever route proves practical will still produce the same evidence as the other targets.

Labs will be easy, but not careless

A major goal is to make labs easy without hiding the sharp edges. The toolkit will help people understand whether Azure SQL Database, SQL Managed Instance, SQL Server on a VM, PostgreSQL, Fabric, or Cosmos DB fits the question they are trying to answer. It will also make security, cost, licensing, data sensitivity, download provenance, and cleanup visible instead of deciding everything silently.

For SQL Server assessment, the plan is to follow Microsoft-supported routes such as SQL Server enabled by Azure Arc and the migration component in SQL Server Management Studio. Compatibility, feature support, security, sizing, cost, and migration readiness will be reported separately.

Fabric is another place where guidance matters as much as automation. People regularly have to choose between a SQL database in Fabric, a Warehouse, and a Lakehouse. I want the Fabric provider to explain those choices in the context of the lab rather than asking the user to pick a product name without understanding the consequences.

That guidance also covers OneLake shortcuts: where the data remains, what the shortcut stores, whose permissions apply, and what freshness means in practice. The provider will guide users through workspaces and items and include a CI/CD route for Fabric. How much of that route can be handled through infrastructure as code remains an open design question that I want to explore.

Some of the first Fabric scenarios I would like to explore are Fabric Unified Admin Monitoring, or FUAM, the Rayfin SDK, and Microsoft's Tabular Model Definition Language, or TMDL. If FUAM is already present, the flow will validate it, explain what it found, and ask before attempting to install or repair anything.

FUAM was originally developed by Gellért Gintli and Kevin Thomas and is hosted in Microsoft's Fabric Toolbox. It is a community accelerator, not an officially supported Microsoft product. Mathias Thierbach created pbi-tools and contributes to TMDL; TMDL itself is a Microsoft language.

Community tools belong in the design

I do not want to build a gallery of anonymous curl commands.

For every catalog item, I need a real home, a maintainer or publisher, a license, a known version, an installation method, and whatever checksum or signing information is available. The entry also needs to say what the toolkit will execute inside the lab.

On the SQL Server side, I already have dbatools in mind. It was created by Chrissy LeMaire and is now maintained by its core team and contributors. I also want to support Brent Ozar's First Responder Kit, including the sp_Blitz family, and Ola Hallengren's SQL Server Maintenance Solution.

For semantic models and Fabric-related work, the list includes Tabular Editor from Daniel Otykier, Measure Killer from Gregor Brunner and the Brunner BI team, pbi-tools, TMDL, FUAM, and Rayfin. A Windows SQL lab may also need the latest SQL Server Management Studio.

These are not all the same kind of project. dbatools and the First Responder Kit are open source. Tabular Editor 2 uses the MIT License, while Tabular Editor 3 is commercially licensed. Measure Killer has free and paid editions but is not an open-source application. pbi-tools uses the AGPLv3 license. Current Terraform releases use Business Source License 1.1; older releases have different licensing. Those distinctions need to remain visible instead of being flattened into the word "free."

The catalog will also accept artifacts that are not public. Users will be able to provide a local file or approved URL, optionally add a checksum, and declare that the artifact or data contains sensitive information. Private software will still receive the same review before the toolkit downloads or executes it.

The third-party notices for Azure SQLVM Toolkit show the direction I want to continue: name the real project, link to its home, credit the people behind it, and be clear about its license.

An open call for projects

If you maintain a database, data generator, assessment utility, administration script, Fabric accelerator, observability solution, PostgreSQL project, or another tool that would make a good lab scenario, please tell me about it. I would like maintainers to use these labs to show what their projects can do.

You can message me on LinkedIn or use the contact form on Kay On Data. For an open-source project, send me the repository, the correct attribution and license, supported platforms, installation details, and the scenario you would use to demonstrate it.

If your project is commercial, please contact me first so we can discuss your idea and what it means to my costs that I'm carrying myself so far.

What comes next

The Azure Data Lab Toolkit repository contains the initial scaffold and the dependency-ordered feature backlog. The work is also visible through GitHub issues and the public project roadmap.

The first milestone is straightforward to describe, even if it is not small: establish the module and provider contracts, then make SQL Server on Azure VM the first fully tested provider from planning through teardown. That includes stronger assessment, cost, testing, and cleanup workflows, as well as the first catalog model for tools, samples, and private artifacts.

Only after that path works from beginning to end will the next provider get its turn. Azure SQL Database and Azure SQL Managed Instance come next, followed by Fabric. Bicep and Terraform will come after the plan and provider contracts are stable enough to give all engines the same meaning.

The larger goal remains simple: make it easier and more cost-friendly to build real data labs, compare platforms, test ideas, install useful tools, load sample or private data, and tear everything down safely afterwards. I also want the toolkit to keep each lab's settings readable and reusable, so the same definition and decisions can become a starting point for real use cases. That is where Azure Data Lab Toolkit is heading.

If you have an idea that I have missed, I would be glad to hear it. You can send me a message on LinkedIn or use the contact form on Kay On Data.