When executing a job, there may instances where the recipes contain steps with errors or warnings. Recipe errors can be introduced in a variety of ways, such as replacing datasets with different columns, changing flow variables, or mistakes from collaborators. These steps can be skipped by execution engines but may result in unwanted data outputs.
Trifacta can alert you in these situations so that you can easily identify and understand any errors and address them. This provides you with better control when maintaining flows over long periods of time.
Note: This feature is available both in Cloud Dataprep and in AWS SaaS editions.
How it works
The option "Ignore Recipe Errors" is available on the Run Job page. You can choose to ignore errors in your recipes and proceed with the job execution.
When this option is selected, a scheduled job may be completed with warning errors, and the details are available on the Job Details page.
Example 1 - Manual Job Runs
In this example, a user introduces recipe errors by replacing a dataset with another one that has missing columns.
When that user runs the job, during compilation the will be validated for recipe errors. A modal dialog will alert the user with a list of any error steps, and the relevant recipes and flows that contain them (including any upstream Reference Datasets).
The user can then address the situation by removing the error steps or fixing the schema at the source. Alternatively, they can choose to continue the execution manually by choosing Continue.
Example 2 Scheduled Job Run
In the next example, a user creates a scheduled job.
If the job was scheduled with ‘Ignore recipe errors’ checked -
The next time the schedule runs, the job will complete successfully, but with warnings. The user is alerted to this situation in the Job Runs history page with a job status of ‘Completed with warnings’.
The user can view the error messages under the Show warnings toggle in the Job Details page which will tell the user where the errors are so they can remedy the situation.
If the job was scheduled with ‘Ignore recipe errors’ unchecked -
When the schedule runs, the job is automatically marked as ‘Failed’ and the user is alerted via email of the Failed job (assuming they haven’t turned off failed job notifications).
The user then clicks on the Job ID link in the email and investigates the errors on the Job Details history page which will tell the user which steps the errors were found in so they can remedy the situation.
Recommended Usage
If you have scheduled jobs running that you would like to fail if any errors are encountered, it is recommended that you uncheck the ignore setting on any scheduled output destinations settings you wish to fail on recipe errors.
Outputs that were created prior to the 8.4 release will have ‘Ignore recipe errors’ checked by default.
More Info