Master Data Extensions

When working with a big amount of data in Salesforce Marketing Cloud, optimization is the key to success.

As you may already know, SQL activities inside Salesforce Marketing Cloud are limited to 30 minutes by default and if they don't finish running before that time, they will return a Timeout error. At this point, you will need to start again from the beginning.

Big companies usually struggle when trying to fit their needs in that timespan, and Master Data Extensions tend to be the solution to this headache.

What are Master Data Extensions?

Master Data Extensions are Data Extensions that contain recurring filters that the marketers need in their campaigns. It can be seen as a combination of relevant source Data Extensions, coming from a CRM, a BI tool, an eCommerce, or somewhere else.

In practice, they are going to work as intermediary Data Extensions where you can format the data so it's more convenient to your purposes, instead of having to look in several Data Extensions for every Target Definition.

Master Data Extensions are going to help you solve two problems at once:

As the Query Activity limit is 30 minutes by default, splitting the queries that you need to do into multiple steps will give you more timespan to complete your segmentation (you'll have 30 minutes to generate the Master Data Extension and 30 minutes to generate your final Data Extension).
It will simplify the day-to-day segment generation: you will keep the complexity on the Master Data Extension generation, and then use them as a simplified view of your data model containing only the information you need.

Define your source Data Model

One of the common mistakes marketers do when defining the data they need in Salesforce Marketing Cloud is to import all the available information in case they would need it at some point. You have to keep in mind that Salesforce Marketing Cloud is not a CRM or a BI system. Hence, it is a marketing automation tool, which means you should only keep data that you are going to need either for defining segments or for personalizing communications.

When defining the data you want to input into SFMC you should answer the following questions:

Do you need this field/Data Extension to generate segments or to modify the content you are going to show your customers?

If your answer is no, you should probably keep that field/Data Extension out of Salesforce Marketing Cloud. Worst scenario, you will be able to add it later if you change your mind.

Do you need all the detailed information contained in this Data Extension, just a portion, or the aggregation calculated from it?

In case you only need a part of the entire data contained in a Data Extension or the aggregations calculated from it, you should apply that filter before updating the data into Salesforce Marketing Cloud.

A good example would be to define the information you need from your transactional data to either filter it and only upload the most recent ones or calculate the aggregations you may need and only upload those calculations (Has bought in the last month?, Has bought X category?).

NOTE: When working with transactional data, you also have to consider the volume of data you are about to import. Salesforce Marketing Cloud is not a data processing tool, so it's better to verify the best way to work with it before updating it to the Cloud. You can reach our support team in case you have any questions about it and we can help you get to the best approach.

How to define the Master Data Extensions I need?

When defining your Master Data Extensions, keep in mind that the main goal is to simplify your daily work, so try to think of daily filters that are usually too complex. These are a couple of common scenarios you may be facing:

Going from Data Extension A to Data Extension D just to get a field from Data Extension D

Imagine you have four Data Extensions (A, B, C, and D) that are all linked as shown in the following picture:

Maybe you are starting your segments from Data Extension A and need a field in Data Extension D, so you would need to do all the way from A to D to get that field. That means combining 4 Data Extensions and reviewing all the data in all of those Data Extensions, which can be a problem in terms of performance.

Instead, you may want to create a Master Data Extension that contains the data from Data Extensions A and D that you need, so you can later access this Data Extension and avoid the extra JOINs.

Master DE with the A data (green) and the D data (blue) needed for this use case

Performing recurring aggregations to filter records

Another common scenario is to query a Data Extension to filter records that fulfill certain conditions. For example, get the contacts that placed more than 3 orders in the last month or get the contacts that opened more than a certain number of emails.

In this scenario, you are going to access the Orders Data Extension, or the _Open, and _Sent Data Views to perform a calculation and then get the records that you want. This can cause a performance issue since you need to access Data Extensions that are too heavy and not very handy.

Instead, the best approach is to create a Master Data Extension with the aggregations and then use it to filter whenever you need it:

For the previous examples:

Get the contacts who placed more than 3 orders in the last month.
- Create a Master Data Extension containing the ID of the contact and number of orders in the last month. Whenever you need to filter out contacts based on the number of orders, you can use this Master Data Extension to avoid accessing the Orders Data Extension and filter them
  recurringly.
Get the contacts that opened more than a certain number of emails.
- Create a Master Data Extension with ID of the contact, number of emails sent, number of opens, and other email engagement information so you can easily access this Master Data Extension in all the Use Cases related to transactional information

Master Data Extensions in DESelect

Once you have defined the Master Data Extensions, all you need to do is create a specific folder for them (so it's easier to find them) and adapt your selections to take them into account.

To make sure that you are always working with updated data, our recommendation is to use the Scheduling of Selections feature to automatically run and refresh your Master Data Extensions during the night, so you have the updated data when you need it.

Also, if you have some recurring selections that you already have scheduled, make sure that the Master Data Extensions are refreshed before then. You can do this by:

Make sure that the schedule of Master Data Extensions is at least 1 hour before the update of your recurring selections.
Creating a Waterfall Selection to update the Master Data Extensions in the first steps and then the recurring selections after.

What affects DESelect performance?

What do you need to know to work with Data Views in SFMC

Can I schedule the running of my selections?

When to opt for a Master DE, a Selection Template, copy the Selection, or Data Sets?