Skip/Notify Processing of Empty files via Azure Data Factory/Synapse

Problem Statement

When it comes to Data Engineering, one of the challenges we face is dealing with Empty or Blank files, or files that only have headers at the source.

So, is there an automated way to either Skip or Notify the processing of these files using Azure Data Factory or Synapse?

Solution

CASE 1: Empty / Blank File

As shown above, the file size for an empty file is 0 B.

Resolution:

We can use the Get Meta Data activity to retrieve the file size. If the file size is ‘0’, we can conclude that it is an empty or blank file.

Output:

CASE 2: File with just Headers

Note: The size of a file with only headers may vary depending on the number of columns in the header.

Resolution:

We can use the Lookup activity to count the number of records (excluding the header) in the file. If the count is ‘0’, we can conclude that it is a file with just headers and no actual data.

Output:

Note: The Lookup activity method can also be used for Case #1 to validate both empty/blank files and files with only headers. It is a better and more efficient solution.