Transform Data in Databricks.

Transform raw data by applying calculations or modifications (e.g., computing TotalPrice) and save the cleaned, structured data to the processed container in ADLS for downstream use.

*

Add transformation logic.

Note: This code is performing a data transformation it creates two new columns inside your DataFrame (df_transformed) based on the existing data.

Creates a new column called FullName by concatenating the values of FirstName and LastName, separated by a space.

Adds another column called Source and fills it with the static value "SampleDB" for all rows.

Copy code:

*

2. Save the transformed data to the processed container.

Confirm Authentication Method

If you’re using Client Credentials OAuth (App Registration + Key Vault), make sure the following configs are set before running the write command:

Note: No mount is needed as long as you’ve configured Spark with the correct credentials. (code below).

Copy code:

*

3. Save the transformed data to the processed container created earlier.

df_transformed.write.mode("overwrite").parquet(
"abfss://processed@datalakestorage000001.dfs.core.windows.net/sales_processed/"
)

*

4. Go back to Azure storage → container → processed and verify that the new data is there.

*

5. Query the processed container.

Copy code:

df = spark.read.parquet("abfss://processed@datalakestorage000001.dfs.core.windows.net/sales_processed/")

display(df)

*

6. Confirm that df (data frame) actually has data.

Copy code:

df.printSchema()
df.show(3)

*

What Happened?

You read raw customer data (from your sample SQL database) into a Spark DataFrame (df).

You then added two new columns:

• FullName → combined FirstName and LastName into one readable name.
• Source → added a constant value "SampleDB" to identify where this dataset came from.

Finally, you wrote the transformed data to your processed container in ADLS in Parquet format.

Note: This step demonstrates ETL (Extract, Transform, Load) using Databricks and stores the clean data for querying.

*

Create Azure Synapse Analytics Workspace.

Note: Synapse provides a serverless environment to query data directly from ADLS.

1. In the portal → Search Azure Synapse Analytics → Create.

*

2. Basics tab Configuration:

• Workspace name: synapse-demo01
• Select the same resource group and region
• Data Lake Storage Gen2: choose existing (datalakestorage001)
• File system: processed
• Click Review + Create → Create.

*

3. Assign the Storage Blob Data Reader role to the Synapse workspace.

*

Query Processed Data in Synapse.

1. Go to Synapse Studio → Develop → SQL Script → New SQL Script → Click publish all

*

2. Run the query to read the first 100 lines from the transformed data:

Copy code:

SELECT TOP 100 *
FROM OPENROWSET(
         BULK 'https://datalakestorage000001.dfs.core.windows.net/processed/sales_processed/',
         FORMAT = 'PARQUET'
) AS [result];

Note: You should see transformed data from Databricks.

Note: This confirms integration between Synapse and ADLS, allowing direct analysis without moving data.

*

Conclusion

In this guide, we walked through an end-to-end Azure data pipeline: from creating a Resource Group and ADLS Gen2 storage, to configuring a Databricks workspace, moving and transforming sample SQL data, and finally querying the processed data in Azure Synapse Analytics.

Key takeaways:

• You learned how to securely connect Databricks to ADLS using Client Credentials OAuth.
• You applied PySpark transformations to enrich and prepare data for analytics.
• You successfully built a pipeline that flows from raw data ingestion to structured data in a data warehouse, demonstrating real-world data engineering practices.
• This workflow highlights best practices for data security, transformation, and cross-service integration in Azure.

Completing this guide equips you with practical, expert-level skills in managing Azure data services and building scalable, secure data pipelines ready for analysis or reporting.

*

Written by Kirill.A - Azure & Cybersecurity Consultant at AntusNet