Databricks Bundle Deployment Script

Замовник: AI | Опубліковано: 20.11.2025

I have eleven existing Databricks jobs that need to be packaged and shipped through the new Databricks Asset Bundles workflow. All code for the jobs is already written in PySpark; what’s missing is a clean, reusable deploy.py that will: • Collect the Python scripts for each job into a single asset bundle • Resolve internal dependencies and set the correct task-level libraries • Push the bundle to my Databricks workspace (Repos or DBFS) • Programmatically create/update the eleven jobs with their respective schedules and cluster definitions The script must rely on PySpark for any data-processing logic that has to run during deployment, and should use the Databricks CLI or REST API (whichever you’re most comfortable with) to handle workspace interactions. I’d like sensible logging so I can trace each deployment step and a concise README explaining how to modify job parameters when a new version of a script is added. Acceptance criteria 1. Running python deploy.py from the repo root uploads the bundle and leaves every job in a RUNNING or SCHEDULED state without manual tweaks. 2. Re-running the same command is idempotent—only changed scripts are redeployed. 3. All eleven jobs execute a small test run immediately after deployment and surface any failures in the console log. Share the finished deploy.py and the README in a Git-friendly structure so I can drop it straight into our repo and start using it.