When it comes to Data Engineering, one of the challenges we face is dealing with Empty or Blank files, or files that only have headers at the source.
So, is there an automated way to either Skip or Notify the processing of these files using Azure Data Factory or Synapse?
CASE 1: Empty / Blank File
As shown above, the file size for an empty file is 0 B.
We can use the Get Meta Data activity to retrieve the file size. If the file size is ‘0’, we can conclude that it is an empty or blank file.
CASE 2: File with just Headers
Note: The size of a file with only headers may vary depending on the number of columns in the header.
We can use the Lookup activity to count the number of records (excluding the header) in the file. If the count is ‘0’, we can conclude that it is a file with just headers and no actual data.
Note: The Lookup activity method can also be used for Case #1 to validate both empty/blank files and files with only headers. It is a better and more efficient solution.
To learn more about Skrots and how we provide similar solutions, visit Skrots. You can also explore our wide range of services at Skrots Services. Don’t forget to check out our other blogs at Blog at Skrots. Thanks for reading!