This article was researched by Arnav Modi, a community user. Arnav is a high school student and aspiring data scientist who spent his summer learning about the SDV and how synthetic data is used to perform ML tasks.

One potential use for synthetic data is to replace real data in the development of new machine learning (ML) models. Imagine a scenario where you need to build a predictive ML model -- perhaps for a function critical to your business, like predicting customer satisfaction or sales success -- with one important consideration: The data is sensitive, so only trusted employees can access it with specific credentials.

Access to sensitive data may create a barrier for a variety of reasons:

• You might not have ML expertise in your organization, which means you need to use external software or contractors to complete the task. However, you are unable to share the data with them.
• Your data is available on a secure, cloud-based platform for trusted employees to access remotely. They work on this data using interactive notebooks. Every time they lose their connection – due to WiFi outages, their laptops falling asleep, etc. – they may lose their work or have to reconnect.
• You have a robust authentication system that your team uses. However, it creates a barrier to entry for rapid, iterative collaboration between members, sharing work and debugging data pipelines. As a result, your collaboration is much slower than it would be if your team could access the data without the need to authenticate.

In cases like this, synthetic data can be an ideal solution: You can create synthetic data based on the original, sensitive data set, and use it more freely during ML development.

One key question will determine if this method succeeds: Is the synthetic data actually useful for your ML task? We performed an experiment to find out.

In the rest of this article, we'll describe our experimental setup and findings. (You can double-check our work in this Colab Notebook.)

Experimental Setup

If an ML model is trained using synthetic data instead of real data, what happens to the model's performance? To answer this question, we identified 3 publicly available datasets (Income, Bank and Airline) that are associated with particular ML prediction tasks. The datasets and tasks are summarized below.

Our experiment compared the performance of an  ML model trained on the original data, vs. one trained on the synthetic data provided by the SDV.

• Control (Original data): How successfully can we complete the ML prediction task if we use the real data? Because some predictions are harder than others, this control helped us identify the overall difficulty of these specific tasks.
• Experiment (Synthetic data): How successfully can we complete the ML prediction task if we use synthetic data instead? We used the SDV's CopulaGAN to generate synthetic data from the three original datasets.

In order to develop and test the ML model, we turned to the SDMetrics library — specifically the ML Efficacy metrics, which build an ML model and evaluate its performance. We used the Binary Decision Tree Classifier and Binary Logistic Regression models. The overall experimental setup is illustrated below.

To obtain reliable findings, we ran 3 iterations and averaged the results.

Results

The graph below shows how well we are able to perform an ML task using the original vs the synthetic data.

Discussion

The original data quantifies the general difficulty of the ML task. Looking at these values, we can see that the Income Dataset is the hardest task, as neither of our methods were able to get above 90% accuracy using the original data.

Comparing the datasets allows us to quantify the suitability of synthetic data for ML development. Our results show a loss of between 1 and 9% of the original efficacy value for all comparisons, with a median loss of roughly 2.5%.

It's important to note that the simplifications we've made for this experiment may be resulting in worse accuracy than we would see in real-world use.

• Applying CopulaGAN out-of-the-box to each dataset is simplistic. In a real-world scenario, the model's parameters would likely be explicitly tuned and constraints would be used to improve synthetic data quality.
• The Decision Tree and Logistic Regression evaluators are relatively simplistic ML classifiers. An ML expert (or ML software) might use more advanced techniques.
• In our scenario, the 3rd party delivers a fully trained, ready-to-go ML model. Another approach is to ask them to use the synthetic data to deliver an untrained model – so that you can train it yourself on the real dataset. This alternative setup, which should increase the prediction accuracy, will be a topic for a future article.

In summary, the accuracy loss we observe represents the worst case scenario. In a production environment, higher-quality ML models and more careful tuning of the SDV will likely minimize performance differences between original and synthetic data.

Takeaways

In this article, we quantified the effect of replacing real data with a synthetic data clone for ML development. Our results show a loss of 2.5% accuracy when using synthetic data. Considering these results, we assess that it is reasonable to explore the use of synthetic data for the purpose of ML development.

In order to maximize the utility of the synthetic data, we recommend tuning the SDV model and using constraints to improve the data quality. In future articles, we'll explore more details about using synthetic data for ML.

Are you using the SDV to solve your ML business needs? Publish your findings on the SDV blog as a guest author! Contact us at info@sdv.dev.

By default, a machine learning model (ML) may not always learn the deterministic rules in your dataset. We've previously explored how the SDV allows user to input their logic using constraints. With constraints, an SDV model produces logically correct data 100% of the time.

While an end user might expect the constraint to "just work," engineering this functionality requires some creative techniques. In this article, we'll describe the techniques we used to build the UniqueCombinations constraint. You can also follow along in our notebook.

!pip install sdv==0.13.1

import numpy as np
import warnings

warnings.filterwarnings('ignore')

What is a Unique Combinations Constraint?

Users frequently encounter logical constraints on the permutations -- mixing & matching -- that are allowed in synthetic data.

To illustrate this, let's use the world_v1 dataset from the SDV tabular dataset demos. This simple dataset describes the population of different cities around the world.

from sdv.demo import load_tabular_demo

data = load_tabular_demo('world_v1')
data = data.drop(['add_numerical'], axis=1) # not needed for this demo
data.head()

Relationship between Name, CountryCode and District

Looking at the data, we can observe that there is a special relationship between the Name of the city, its CountryCode and its geographical District: When generating synthetic data, the model should not blindly mix-and-match these values. Instead, it should reference the real data to verify whether the combination is valid. This is called a UniqueCombinations constraint.

For example, take a particular city, like Cambridge, which appears 3 times in our dataset.

data[data.Name == 'Cambridge']

The constraint states that Cambridge should only ever appear with GBR (England), CAN (Ontario) or USA (Massachusetts). It is invalid if it appears in any other region -- for eg. Cambridge, France.

How does the SDV handle a Unique Combination out-of-the-box?

Let's try running the sdv as-is on the dataset to see what happens. We'll use the GaussianCopula model on our dataset.

from sdv.tabular import GaussianCopula

np.random.seed(0)

model = GaussianCopula(
  categorical_transformer='label_encoding' # optimize speed
) 
model.fit(data)

Now, let's generate some rows to inspect the synthetic data.

np.random.seed(12)
model.sample(5)

Although the sdv is generating known city names, countries and districts, their combinations don't make sense. We can also go back to our original example and generate only some rows for Cambridge.

np.random.seed(10)

conditions = {'Name': 'Cambridge'}
model.sample(5, conditions=conditions)

The result is a variety of Cambridges that aren't necessarily in USA, GBR, or CAN. These aren't valid cities!

What's going on? The SDV models include probabilities that some unseen combinations are possible. This is by design: Synthesizing new combinations -- that don't blatantly match the original data -- helps with privacy.

However in this particular case, we aren't worried about the privacy of a city belonging to a country or district. We actually do want the data to match. This is why we need to build a constraint.

Fixing the data using rejecting sampling

In our previous article, we described a solution called reject_sampling that works on any type of constraint and is very easy to build: We simply create the synthetic data as usual and then throw out (reject) any data that doesn't match.

In theory, this can solve our UniqueCombinations constraint. In practice, this strategy is only efficient if the model can easily generate acceptable data. Let's calculate the chances of getting an acceptable combination (Name, CountryCode, District) from the model.

np.random.seed(0)

# Sample data from the model
# The sample may include combinations that aren't valid
n = 100000
new_data = model.sample(n)

# Calculate how many rows are valid
combo = ['Name', 'CountryCode', 'District']
merged = new_data.merge(data, left_on=combo, right_on=combo, how='left')
passed = merged[merged['ID_y'].notna()].shape[0]

# Print out our results
print("Valid rows: ", (passed/n)*100, "%")
print("Rejected rows: ", (1 - passed/n)*100, "%")

Valid rows:  0.038 %
Rejected rows:  99.96199999999999 %

With such a low probability of passing the constraint, this strategy can become intractable.

Fixing the data using transformations

A more efficient strategy is for the ML model to learn the constraint directly, so it always produces acceptable data. We can do this by transforming the data in a clever way, forcing the model to learn the logic.

Our previous article described how to do this for a different constraint. Unfortunately, the exact same transformation won't work to solve our current UniqueCombinations constraint. The transform strategy requires a different, creative solution for each constraint. So we have to start from scratch.

Can you think of any other ways to enforce UniqueCombinations?

A solution: Concatenating the data

One solution is to concatenate the data. That is, rather than treating the city Name, CountryCode and District as separate items, we treat them as a single value. This will force the model to learn them as 1 single concept rather than as multiple columns that can be recombined.

Let's see this in action.

# create transformed data that concatenates the columns
data_transform = data.copy()

# Concatenate the data using a separator
data_transform['concatenated'] = data_transform['Name'] + '#' + data_transform['CountryCode'] + '#' + data_transform['District']

# We can drop the individual columns
data_transform.drop(labels=['Name', 'CountryCode', 'District'],
                    axis=1, inplace=True)

data_transform.head()

Now, we can train the model using the transformed (concatenated) data instead.

np.random.seed(35)

# create a new model that will learn from the transformed data
model_transform = GaussianCopula(categorical_transformer='label_encoding')
model_transform.fit(data_transform)

# this will produce transformed data
output = model_transform.sample()
output.head(5)

To get back realistic-looking data, we can convert the concatenated column back into Name, City and District.

import pandas as pd

# Split the conatenated column by the separator and save the reuslts
names = []
countrycodes = []
districts = []

for x in output['concatenated']:
  try:
    name, countrycode, district = x.split('#')
  except:
    name, countrycode, district = [np.nan]*3
  names.append(name)
  countrycodes.append(countrycode)
  districts.append(district)

# Add the individual columns back in
output['Name'] = pd.Series(names)
output['CountryCode'] = pd.Series(countrycodes)
output['District'] = pd.Series(districts)

# Drop the concatenated column
output.drop(labels=['concatenated'], axis=1, inplace=True)

As a result, the output now looks like our original data.

output.head()

Most importantly, the Name, CountryCode and District columns now make sense!

Caveats of transforming the data

The transform strategy is an efficient and elegant approach to modeling. But there is a downside: The transform strategy might lose some mathematical properties.

To see why, consider the model's perspective:

• Cambridge#GBR#England is completely different from
• Cambridge#USA#Massachusetts is completely different from
• Boston#USA#Massachusetts

The problem is that two of these actually have something in common -- they are located in Massachusetts, USA. So the model will not be able to learn anything special about Massachusetts or USA as a whole.

As an example, let's see how well the model was able to learn populations of US-based cities.

import matplotlib.pyplot as plt

# Populations of real US cities
real_usa = data.loc[data['CountryCode'] == 'USA', 'Population']

# Populations of synthetic US cities
synth_usa = output.loc[output['CountryCode'] == 'USA', 'Population']

# Plot the distributions
plt.ylabel('US City Data')
plt.xlabel('Population')
_ = plt.boxplot([real_usa, synth_usa],
                showfliers=False,
                labels=['Real', 'Synthetic'],
                vert=False
)
plt.show()

The real data shows less variation in city population than the synthetic data. The differences make sense because our model wasn't able to learn about the USA as one complete concept.

Can we fix this? It's challenging to fix this issue without degrading the mathematical correlations in some other way. If you have any ideas, we welcome you to join our discussion!

Inputting a UniqueCombination into the SDV

We built the constraint -- both the reject_sampling and transform approaches -- directly into the SDV library. If you have sdv installed, this is ready to use. Import the UniqueCombinations class from the constraints module.

from sdv.constraints import UniqueCombinations

# Create a Unique Combinations constraint
unique_city_country_district = UniqueCombinations(
  columns=['Name', 'CountryCode', 'District'],
  handling_strategy='transform' # you can change this 'reject_sampling' too
)

# Create a new model using the constraint
updated_model = GaussianCopula(
  constraints=[unique_city_country_district],
  categorical_transformer='label_encoding'
)

Now, you can train the model on your data and sample synthetic data.

np.random.seed(35)

updated_model.fit(data)
updated_model.sample(5)

All of the synthetic data is guaranteed to follow the UniqueCombinations constraint.

Takeaways

1. We can identify a UniqueCombinations requirement by asking: Should it be possible to further mix-and-match the data?
2. We can enforce any logical constraint by using reject sampling, which throws out any invalid data. This is not efficient for UniqueCombinations.
3. An alternative approach is to transform the data, forcing the ML model to learn the constraint. For UniqueCombinations we transformed the data by concatenating it.
4. The logic for UniqueCombinations is already built into the SDV's constraints module, and is ready to use.

Further reading:

• Engineering Constraints Blog Article
• Handling Constraints User Guide
• Tabular Constraints API

We started SDV open source in 2018 at MIT with the goal of creating a powerful, usable, machine learning-based synthetic data generation software system. The core belief that drove us was the conviction that more than 90% of data work can be done using synthetic data instead of real data. Early experiments at MIT had been promising and we were ready to invest our time and energy into that promise.

Now, 3 years later, we are pleased to see that the market demand for synthetic data is increasing. In a 2021 article, Gartner predicted that 60% of data used for AI & analytics will be synthetic by 2024.

As time progressed, we used feedback from our users to make numerous improvements to the SDV (see articles Part 1 and Part 2). In response, we've seen increased usage, validating the market need for synthetic data generation software. In this article, we'll describe the SDV growth trends in detail.

Persistent 4x/year growth in downloads

Every year we are experiencing a 4x increase in SDV downloads. In 2021, we had 135,000 downloads of SDV – up from 30,576 in 2020. From the start of 2020 to the end of 2021, we have seen 16x total increase in SDV downloads. The figure below shows our yearly usage.

The downloads are coming from all over the world. In the map below, we list the top 10 countries.

Why are users downloading the SDV? We know that they want to create synthetic data, but they are using the synthetic data to solve a variety of different needs. We will explore this more and share it in a future article.

Over a thousand new community members

Another measure of our growth – and validation from the market – comes from the SDV community we've built on our GitHub and Slack. In 2021, we welcomed more than 1000 new members to these spaces.

As this article points out, members contribute in several different ways: Many help increase awareness of an open source solution for this enterprise pain point. Meanwhile, others jump in, use it and give feedback actively. In 2021, we doubled the number of unique users raising issues on our GitHub. Throughout the  year, over 200 members actively participated in our forums by raising GitHub issues or contributing to discussions on Slack.

Enterprise feedback is particularly useful to us. This type of feedback comes from users who are solving targeted business problems with the SDV. Direct and succinct feedback explains what would make the SDV more useful. An example is shown below.

Our team addresses the user feedback throughout the entire SDV ecosystem. The ecosystem includes not only modeling, but also the ability to compare models through SDGym and measure synthetic data quality through SDMetrics. In 2021, the team put out 49 releases throughout the SDV ecosystem, doubling our number of releases in 2020.

Looking forward to 2022!

We are looking forward to 2022! With so many users giving us feedback, we have a long list of features that we want to incorporate. We can't wait to share with our community what everyone is using SDV for, and keep on climbing to our original goal: 90% of data work accomplished with synthetic data.

The SDV uses machine learning (ML) to automatically learn rules (aka correlations) from real data and generate accurate synthetic data. While these models are powerful, they may not learn everything. In our previous article, we described how the SDV models may not learn deterministic rules. These are patterns and laws that are inherent to the dataset:

• They are unchangeable, no matter what data you input.
• They describe rules that must apply to every row, no exceptions.

Luckily, it's possible for you to improve the machine learning model: When you input constraints, it ensures the model will learn deterministic rules and ultimately improve the quality of your synthetic data.

In this article, we'll dive into the technical details of how you can apply constraints and how they work under-the-hood. You can also follow along in our notebook.

!pip install sdv==0.13.0

import numpy as np
import warnings

warnings.filterwarnings('ignore')

The Dataset

The dataset we're using comes from a Kaggle Competition hosted by Expedia. We've modified the data slightly for our use.

from sdv.demo import load_tabular_demo

data = load_tabular_demo('expedia_hotel_logs')

In this real-world dataset, each row represents a search result for a hotel booking.

For the purposes of this notebook, we'll drop some columns that aren't useful to us.

import pandas as pd

# Drop some columns that aren't useful for this demo
drop_columns = ['date_time', 'user_location_country', 'user_location_region',
                'user_location_city', 'user_id', 'srch_destination_id',
                'hotel_country', 'hotel_market', 'hotel_cluster',
                'srch_destination_type_id', 'orig_destination_distance',
                'posa_continent', 'site_name', 'channel']
data = data.drop(drop_columns, axis=1)

# make sure these columns are read as datetimes
for col in ['srch_ci', 'srch_co']:
  data[col] = pd.to_datetime(data[col])

# Inspect the data
data.head()

The search parameters, for finding a hotel room, saved in this dataset come from from user's input. For example:

Deterministic Rule

In order for the search to be valid, the searched check-in date must happen before the searched check-out date. That is: srch_ci < srch_co.

This is an inherent property of any search, not just for this particular dataset -- we call this a deterministic rule. We can verify if this is true by checking for any exceptions.

print('Violations of the deterministic rule')
len(data[data['srch_ci'] > data['srch_co']])

0

Will SDV's machine learning model learn this out of the box?

To test this, let's use SDV to learn a GaussianCopula model from the data and sample synthetic data.

from sdv.tabular import GaussianCopula

np.random.seed(0)

model = GaussianCopula(primary_key='log_id')
model.fit(data)

synth_data = model.sample(500)
synth_data.head()

Now, we can inspect the synthetic data to see if there are any invalid rows.

invalid_row_indices = synth_data['srch_ci'] > synth_data['srch_co']
invalid_rows = synth_data[invalid_row_indices]

num_invalid = len(invalid_rows)
perc_invalid = num_invalid / len(synth_data) * 100
print('Number of invalid rows:', num_invalid, '(', round(perc_invalid, 2), '%)')

invalid_rows.head()

The majority of the rows (94.8%) are valid, meaning the model learned the rule pretty accurately. It learned probabilistically that if the srch_ci is higher srch_co should be even higher. However, some invalid rows (~5%) are still created so the model did not learn this deterministic rule.

This raises the question: What can we do to enforce a deterministic rule?

Improving the synthetic data

Let's explore some options for enforcing our deterministic rule in order to improve the overall quality of the synthetic data.

Rejecting invalid data

The simplest solution is to simply drop the invalid rows, and continually sample from the model until the desired amount of valid rows are produced. We call this reject sampling.

The code below performs reject sampling until we have synthesized 500 rows.

import pandas as pd

# Keep track of how many valid rows we've sampled
num_valid_rows = synth_data.shape[0] - invalid_rows.shape[0]

while num_valid_rows < 500:
  # Reject the invalid data 
  synth_data = synth_data.drop(invalid_rows.index)
  
  # Create new data to replace the invalid data
  new_data = model.sample(500-num_valid_rows)
  synth_data = pd.concat([synth_data, new_data])
  invalid_rows = synth_data[synth_data['srch_ci'] > synth_data['srch_co']]
  num_valid_rows = synth_data.shape[0] - invalid_rows.shape[0]

synth_data.reset_index(drop=True, inplace=True)

Now, there are no invalid rows in our dataset.

invalid_rows = synth_data[synth_data['srch_ci'] > synth_data['srch_co']]
invalid_rows.shape[0]

0

In this example, we got lucky. Only a small percentage of the rows were invalid each time sample was called.

What would happen if majority of the rows were invalid every time we sampled? It would take a longer time to get all the desired rows. Sampling time is the primary drawback of reject sampling. Is there another approach we can use to improve the time?

Transforming your data

Instead of reject sampling, what if the model never produced invalid rows in the first place? To achieve this, we can alter the input data to the model so it's forced to learn the constraint.

Let's stop giving the srch_ci and srch_co to the model. Instead, let's teach the model to learn the srch_ci and the difference between the dates.

difference = srch_co - srch_ci

The model will produce srch_ci and difference as a result. Then, we can re-compute srch_co with the opposite formula.

srch_co = srch_ci + difference

(Of course, we need to make sure the difference is always positive, which we can do using a log + 1.)

Let's see this in action.

# Compute the difference
diff = (data['srch_co'] - data['srch_ci']).astype('timedelta64[D]')

# Take the log and add one to ensure that it's positive
date_diff = np.log(diff + 1)

# The model should learn this column instead of the checkout date
modified_data = data.drop('srch_co', axis=1)
modified_data['difference'] = date_diff
modified_data[['srch_ci', 'difference']].head()

Now, we can fit the model with the modified data. The new samples will include the srch_ci and date_diff columns.

np.random.seed(20)

modified_model = GaussianCopula(primary_key='log_id')
modified_model.fit(modified_data)

modified_synth_data = modified_model.sample(500)
modified_synth_data[['srch_ci', 'difference']].head()

We can recompute the srch_co based on srch_ci and difference.

# Undo the log+1 that we added
diff = (np.exp(modified_synth_data['difference'].values).round() - 1).clip(0).astype('timedelta64[ns]')

# Reconstruct the end_date and remove the date_diff column
modified_synth_data['srch_co'] = modified_synth_data['srch_ci'] + diff
modified_synth_data = modified_synth_data.drop('difference', axis=1)

modified_synth_data.head()

Let's verify that this computation does not create any invalid rows.

invalid_rows = modified_synth_data[modified_synth_data['srch_ci'] > modified_synth_data['srch_co']]
invalid_rows.shape[0]

0

The transformation worked! In our case, this was a more efficient way to enforce the deterministic rule.

But if our rule were more complex -- and we couldn't think of a transformation -- we could always fall back to reject sampling.

Inputting deterministic rules in the SDV

We've seen how reject sampling and transform can be used to improve the quality of the synthetic data by accounting for deterministic rules. However, it may be cumbersome for you to manually implement these strategies. In fact, we saw some common problems in our SDV user community:

• Users had multiple deterministic rules in their dataset. For example, there could be multiple comparisons between different pairs of columns.
• Users from multiple domains often had the same kind of deterministic rule. For example, one column being greater than another is a common deterministic rule, agonistic of a use case or domain.

To solve these problems, we introduced a constraints module in the SDV. With the constraints module, SDV users can easily input deterministic rules. Let's look at an example.

Using the SDV constraints module

The constraints module in the SDV contains several different types of pre-defined deterministic rules.

We will use the GreaterThan constraint, which will enforce that one column's values are always greater than another's.

from sdv.constraints import GreaterThan

Next, we can input the logic of our deterministic rule by creating a constraint object. The GreaterThan constraint accepts the column names as input.

gt_constraint = GreaterThan(
  low='srch_ci',
  high='srch_co')

Finally, we can input this constraint when instantiating the model.

np.random.seed(10)

# Apply the constraint to the model
model_with_constraint = GaussianCopula(
  primary_key='log_id',
  constraints=[gt_constraint])

model_with_constraint.fit(data)

# Sample synthetic data
constrained_data = model_with_constraint.sample(500)
constrained_data.head()

As a result, we should see that all 500 generated rows are valid on the first try. No invalid rows are present in our dataset.

invalid_rows = constrained_data[constrained_data['srch_ci'] > constrained_data['srch_co']]
invalid_rows.shape[0]

0

Using the SDV was much simpler than writing the code ourselves! Plus, we can create multiple constraints for the same dataset an easily use them on other datasets.

Specifying the strategy in the constraints module

By default, the GreaterThan constraint uses the transform strategy. However, you can use the handling_strategy argument to control this. This argument accepts 'reject_sampling' or 'transform' as valid strategies.

gt_reject_constraint = GreaterThan(
  low='srch_ci',
  high='srch_co',
  handling_strategy='reject_sampling' # specify the strategy
)

Similar to before, we can then input this constraint into the model.

np.random.seed(30)

# Apply the constraint to the model
model_with_reject_constraint = GaussianCopula(
  primary_key='log_id',
  constraints=[gt_reject_constraint])

model_with_reject_constraint.fit(data)

# Sample synthetic data
constrained_reject_data = model_with_reject_constraint.sample(500)
constrained_reject_data.head()

invalid_rows = constrained_reject_data[constrained_reject_data['srch_ci'] > constrained_reject_data['srch_co']
invalid_rows.shape[0]

0

What other deterministic rules are already available in SDV?

The GreaterThan constraint is one kind of deterministic rule, but there may be others that apply to your dataset. The SDV offers more constraints for other types of logic.

• Unique when values in a column must be unique to the entire dataset.
• UniqueCombinations to limit the permutations between multiple columns.
• Positive and Negative to enforce boundaries.
• ColumnFormula when there is a formulaic association between columns.
• Rounding to enforce decimal precision.
• Between when one column's values must be between 2 other values.
• OneHotEncoding when your data includes a variable with one hot encoding.

For each of them, you can specify handling strategies for reject_sampling to discard invalid data or transform to modify the data (unique to each constraint).

What if my rule isn't included in the module?

You may come across a rule that cannot be described by any of the constraints classes in the SDV. In this case, you can define a CustomConstraint with logic specific to your use case.

Additionally, consider filing a feature request on GitHub with details about your use case & scenario. We can add your logic as a pre-defined constraint so others can benefit from it too!

Takeaways

In this notebook, we explored what happens when we have a deterministic rule in our dataset.

1. Machine learning models may not able to learn the deterministic rules out of the box, but it is possible to improve the model to learn these types of rules.
2. Deterministic rules can be handled by discarding invalid data (reject sampling) or by adding some clever preprocessing to your code (transforming).
3. The SDV offers a constraints module that allows you to input commonly found deterministic rules. You can specify the handling strategy for each constraint and apply multiple rules to the same dataset.

Further Reading

For further information about constraints refer to the Handling Constraints User Guide.

In our previous article, we explored how machine learning (ML) plays a key role in synthetic data creation. One of the biggest strengths of ML is automatic rule detection (also known in ML terms as correlations): The algorithms are designed to learn patterns in the data, even without additional user input. The result is synthetic data that resembles the original, right down to its mathematical properties!

However, in some cases, applying an ML model right out of the box may not immediately achieve the desired result. In this article, we'll explore the strengths of ML models and go through those areas where user input may be required.

Strengths of ML Models

The goal of any ML-based synthetic data generation software is to learn from and emulate the input data. To illustrate this, let's pretend you work in the car insurance business, and you're in possession of a real dataset related to drivers and their insurance:

An ML-based system, such as the Synthetic Data Vault (SDV), will learn patterns from the real data and use it to create new synthetic data. Recall some of the important patterns that ML algorithms detect:

• Shapes. The general shape of the data. For example, in the dataset above, 50% of drivers have Collision Coverage and the Annual Premium is uniformly scattered between $3,000 and $9,000.
• Correlations. The trends within the data. For example, having Collision Coverage -- especially Standard coverage -- means a higher Annual Premium.

These shapes and correlations will be present in the synthetic data that is outputted by the ML model, as shown below.

Perhaps the single biggest strength of an ML algorithm is its ability to learn rules by looking for general patterns in the data, using probability and statistics.

What ML models do not learn out of the box

Let's take a closer look at the synthetic car insurance data. You might notice that two of the rows in the synthetic data don't make complete sense. Below, we've highlighted the errors.

Do you see what has gone wrong? In the first row, the license expired 3 years earlier than it was issued. In the last row, a driver without Collision Coverage has a Collision Policy Type. Additionally, the same Customer ID has been repeated in Row 3 and Row 4.

There are three rules that the ML algorithm did not follow:

1. License Expiration > License Issue Year
2. If Has Collision Coverage = NO, then Collision Policy Type must be empty
3. All Customer IDs must be unique

Why does the ML model easily pick up on some rules and not others? To answer this question, we can look closely at the rules themselves. All of the rules that the ML model successfully learned -- including the distribution shapes and the correlations -- were based on general trends. These probabilistic rules apply to a majority of the relationships within the dataset, but not all of them. Although they have to make sense in aggregate, a few rows may be exceptions.

By contrast, the rules that the ML model failed to learn were stricter. These deterministic rules describe intrinsic laws of nature, time or logic. Each and every row must adhere to them, and they won't change regardless of  how much (or how little) data has been given to the ML model.

To continue with the driving theme: A probabilistic rule is like a yield sign, signaling a general recommendation that works out differently for each individual situation -- some cars may need to stop, while others just slow down. Meanwhile, a deterministic rule is like a stop sign, demanding that every single car must come to a full stop.

By default, our ML model assumed that all rules were probabilistic. When this happens, synthetic data still generally follows the desired properties -- for example, License Expiration > License Issue Year -- for most of the rows, but not for every row.

Improving the ML Models using constraints

Just because the ML model didn't automatically follow a deterministic rule doesn't mean that it can't. It's possible to improve the model so that it understands this type of rule. As a user working with the SDV, you can input deterministic rules into your model using constraints.

An ML model built using constraints will accommodate both probabilistic and deterministic rules.

Do you need SDV constraints?

Deterministic rules are often easy to spot in your dataset: They are the rules that every single row must follow in order to be valid, regardless of how much data there is overall.  But even if you identify the right constraints, there are some cases where you might not actually want to supply them to the SDV.

Because the SDV learns probabilistic rules, most of the synthesized data is generally valid. Having a few errors sprinkled in might actually be beneficial if you want your synthetic data to cover some edge cases. For example, if you're using the synthetic data to test insurance claim software, leaving in some weird data points might help you ensure that the software can handle unexpected cases -- like the License Expiration accidentally being set too early.

The figure below shows a few questions you can ask to determine whether adding a constraint is the right approach.

The SDV Constraints offering

If you decide that adding deterministic rules is important for generating your synthetic data, the SDV has many different constraints to choose from! The table below describes the constraints you would need in order to define the deterministic rules that would best mold your Car Insurance dataset.

The SDV offers many more possible constraints, including:

• UniqueCombinations
• Positive and Negative
• Rounding
• Between
• OneHotEncoding

You can add multiple constraints to the same dataset in order to accommodate all the deterministic rules you need. For more details, read the Constraints User Guide.

Takeaways

In this article, we learned that:

• Data is governed by rules. The SDV automatically learns probabilistic rules, which describe overall trends or patterns in the data.
• However, sometimes the data has deterministic rules, which are always inherent no matter how much or how little data there is. ML-based systems, including the SDV, may not enforce deterministic rules out of the box.
• Users can input deterministic rules to the SDV using constraints. To figure out whether you should input a constraint, ask yourself whether there are any rules that the data must always follow. There are many constraints to choose from.

In future articles, we'll dive deeper into this topic. We'll explore the technical details behind constraints, and how exactly the SDV's ML models are able to learn deterministic rules.

Data is a great source of information. Real data — which is based on observations of real-world phenomena like weather, movements on a factory floor or the activities of a user base — can help us notice trends, increase business efficiency and solve problems.

But data can be helpful even if it isn’t real. This data, sometimes called fake or test data, doesn’t come directly from real-world observations, but is instead artificially crafted by a human or machine. The latest and most complex iteration of this data type — what we call synthetic data — builds on previous work done in this space.

In this article, we'll go through the history of fake data. By the end, you'll be able to answer the following questions:

• What were the original motivations and tools for manually creating data?
• What differentiates synthetic data from other types of fake data?
• What role does machine learning play in generating synthetic data?

The Dawn of Fake Data: Test Data Management

One group of people has worked with fake data for a long time: software engineers. They need data in order to test the systems they build, and the real stuff isn't always usable (for example, due to privacy).

Let's pretend it's the early 2000s, and you're an IT professional working at a bank. You're responsible for the software that updates account balances after each transaction. You'd like to test this software before putting it into production. What do you do?

Most likely, you'll come up with a few test scenarios to ensure that your functionality — updating the balance — can properly handle a variety of inputs.

Notice that in order to create these scenarios, you had to generate data: various starting balances ($500, $20, $10) as well as a transfer amount ($20). This is an early version of using fake data in order to test your software!

Using Tools for Manual Creation

Now let's fast forward in time. Over the years, your software has gotten even more complex, and you're constantly adding new functionalities. For example, maybe you start allowing transfers with foreign currency.

You need to test these functionalities before you roll them out. To save time, you might end up using -- or creating -- a tool that allows you to generate and manage fake data for testing.

The simplest tool may be a basic permutation, as illustrated below.

A more sophisticated tool might allow you greater control over the rules the data must follow. It will also allow you to create more columns as your functionalities increase. For example, maybe the bank now offers two different account types: Premium and Normal.

Now you need a test data generation tool that can handle all of these variables and come out with something like this:

Many test data management tools use sophisticated logic to precisely create these data columns and their values. But the rules they use are manually written, and rely on human intuition and domain knowledge. For example:

• Account type = Premium 10% of the time and Normal 90% of the time
• Starting balance is between $10,000 and $250,000 if Account type = Premium
  or between -$1,000 and $10,000 if Account type = Normal
  
• Transfer amount follows a bell curve with a mean of $7,500 and standard deviation of $1,000
• Etc.

There are downsides to this manual approach. It takes time and effort to come up with these rules, to keep track of them, and to update them as your application changes.

Adding Machine Learning

Adopting machine learning (ML) opens up entirely new avenues in data generation. In the process, it gets rid of some of these downsides.

At a high level, ML-based software (such as the Synthetic Data Vault) works in three steps:

1. The user inputs real data into the ML software
2. The ML software automatically learns patterns in the data
3. The software outputs data that contains those patterns

Let's go back to our banking example to see how this works. It's now 2021 and you're using the SDV to generate your test data. You input all the transactions your bank has handled in the last week.

After modeling, the SDV outputs entirely new data that looks and behaves like the original. An illustration of this is shown below.

Notice that the output data contains many of the same properties as the original. The model learned all of the following information:

• Ranges & Categories. Transfers range from $5K to $10K. Bank accounts can be either premium or normal. Etc.
• Shapes. 10% of accounts are premium. Transfers follow a bell curve distribution with a mean of $7,500 and a standard deviation of $1,000. Etc.
• Correlations. Premium bank accounts tend to have higher balances ($10K to $250K) than normal accounts (-$1K to $10K).

In other words: while the old test data management tools required you to manually come up with rules, ML-based tools learn these rules automatically. Moreover, they can learn new information. For example, the ML picked upon a couple of extra correlations:

• Premium accounts are more likely to transfer foreign currency.
• Normal accounts are more likely to be overdrawn (transfer more than their current balance).

Using an ML-based data generation tool will help you ensure that your software is robust against these typical cases. And while manual data generation tools generate fake data, ML-based approaches generate what we call synthetic data.

Benefits of Synthetic Data

There are some clear advantages to using synthetic data over fake data, especially in software testing. Below, we've detailed a few.

• Saves time with automation. Because ML automatically learns patterns from the real data, there is no need to spend a lot of time coming up with and inputting rules. ML learns rules that you may even miss.
• Is usable by non-experts. Realistic fake data can only be generated by domain experts, who know the precise rules governing the dataset. However, anyone can generate synthetic data. All they have to do is input the real data and the ML software takes care of the rest!
• Increases adaptability. Applications and data will inevitably change over time. It's easy to update synthetic data as this happens, simply by retraining the ML model with newer data.

Benefits of synthetic data expand beyond software testing. The SDV Community is using synthetic data for an ever-increasing variety of tasks, including machine learning development, de-biasing datasets and scenario planning.

Key Takeaways

In this article, we surveyed numerous ways of creating and using data  that is not real. In particular, we learned that:

• Creating fake data is not a novel concept. Older generations of tools will output fake data when given an explicit list of rules. This is especially useful for software testing.
• Adding ML to this process is a newer evolution. Users input real data into the ML model, and it's able to automatically infer the rules. Data generated using ML-based systems is known as synthetic data.
• Synthetic data's key advantages include its automation and adaptability. The uses of synthetic data expand beyond software testing.

In future articles, we'll put ML models to the test! We'll uncover their strengths and weaknesses, and guide you through getting the most from synthetic data using the Synthetic Data Vault.

The Synthetic Data Vault (SDV) is a software system that allows users all over the world to input a dataset and generate synthetic data. The SDV was born out of academic research at MIT — but in 2018, we open-sourced it, so that people all over the world could use it.

Since then, we've been listening carefully to our community's feedback, making sure that we address any gaps between theoretical academic research and practical use. This article is the second in a multi-part series detailing recent improvements to the SDV that make it work in the real world. Here we'll discuss how we've amped up the data synthesis workflow. (For our previous discussion about how we've improved core models, see Part 1.)

What are workflows?

We open sourced the SDV not just to let users generate synthetic data, but also to allow them use that data to solve real-world problems. Our community taught us that actually using the SDV involves a multi-step process — and that improving the system means paying attention to this entire workflow, not just the core machine learning.

According to our users, this workflow boils down to a few generalizable steps:

1. Identifying real datasets that need to be synthesized
2. Transforming the datasets into a machine-readable format
3. Running the machine learning model
4. Synthesizing data according to particular specifications
5. Reversing the transformations such that the synthesized data looks like the original
6. Evaluating the synthesized data that results

These steps are illustrated in the diagram below.

The key insight from our users was that the application of machine learning models is only one step of a much larger puzzle. When the open source community helped us understand this, we were able to improve on the SDV software by adding in transformations, synthesizing options, and evaluation tools -- all detailed below.

Transforming Data

One major lesson from our open source community was how messy real-world datasets are compared to those used in academia. Academic datasets often come pre-sanitized and ready for numerical use. In the real world, however, databases are growing and changing constantly, and are often significantly different from the optimal yet theoretical structures used by machine learning researchers.

Two thorny data types frequently encountered in the real world are datetimes and null values.

• Datetimes can follow many different formats, including YYYY-MM-DD or MM-DD-YY. However, machine learning models accept numerical values only. Usually these are Unix timestamps, defined as the number of seconds that have elapsed since January 1, 1970. By this logic, a date like 2021-01-01 will transform into the number 1609488000.
• Null values also present a problem for mathematical models when they appear in numerical data. While users can tell models to ignore these values, the presence of a null might actually indicate something important, like a user declining to answer a question. To account for this, the SDV creates a new, binary column to address whether the original value is null.

To solve this problem, we introduced a new library called Reversible Data Transforms (RDT). The RDT library contains necessary logic for transforming different types of real world data to its machine-ready counterpart — as well as the logic for its reversal, so that a synthetic data user won't know the difference. The RDT is a standalone library that can reach beyond the synthetic data space, helping data scientists and academics across fields to clean their data. Since November 2020, the RDT has been supported on all major platforms including MacOS, Windows, and Linux.

Synthesizing Data Conditionally

When we first imagined the SDV, we assumed users would simply want to use all the synthetic data generated by the model. However, we soon found that some users have more complex needs, and require more control over the data they synthesize — opening up new possibilities for synthetic data in the process.

For example, one of our users, an engineer, found a whole new use for SDV. The engineer was writing a machine learning classifier on a dataset when they noticed it was unbalanced. Applying any algorithms to this dataset would lead to biased models. The engineer realized that, if used strategically, SDV could actually debias the data — if it only generated data with rarer attributes, the synthetic data it created could be combined with the real data to form a fully balanced dataset.

In February of 2021, we added conditional sampling to the SDV to enable this use case. Now, users can specify attributes or values that must be present in the synthesized data. In addition to debiasing datasets, users can use this feature to test hypothetical scenarios.

Evaluating Synthesized Data

When the entire system is working smoothly and outputting synthetic data, users still need to know: Is the data good enough to use? This vital question inspired us to add evaluation capabilities to the SDV. In doing so, we faced two key challenges: Defining the metrics, and creating a useful process.

Metrics

No single metric perfectly captures the different dimensions of synthetic data users may want to evaluate. Some want to preserve a high degree of mathematical likeness, others want to emphasize a particular column for machine learning predictions, and still others are more focused on threat models that can compromise privacy.

To address this, we created a separate library, SDMetrics, to define evaluation metrics. The library now includes a suite of metrics that cover differentiation of synthetic and real data, statistical likeness, and privacy.

Application

Rather than apply metrics on an ad-hoc basis, some SDV power users were creating mini-workflows to rapidly test out different models, datasets and evaluation criteria in succession. Inspired by their innovation, we created SDGym, a system that allows users to input models, datasets and success metrics to build a comprehensive evaluation framework.

The SDV Software Today

The SDV software is continuously evolving based on community feedback. In this article, we discussed improvements to the workflow surrounding synthetic data generation, including data transformations, sampling methods and evaluation tools. Earlier, in Part 1 of this series, we discussed the core synthetic data models themselves. In future blog articles, we plan to dig deeper into each of these areas, and to uncover new ones with you.

Like the SDV, this blog is a collaborative effort. Use our Slack to let us know which topics you'd like to hear more about. And as always, use GitHub to file technical issues with the system. Working together, we can make SDV the most trusted, transparent and comprehensive platform for synthetic data generation!

For other inquiries, please contact info@sdv.dev.

In our last post, we introduced the Synthetic Data Vault (SDV) — a software system that allows users to input a dataset and generate synthetic data. The SDV was born out of academic research at MIT — but in 2018, we open-sourced it, so that people all over the world could use it.

Since then, we've been listening carefully to our community's feedback, making sure that we address any gaps between theoretical academic research and practical use. This multi-part series details recent improvements we've made so that SDV works in the real world. In this article, we focus on the machine learning-based modeling techniques that form the core of the system, while Part 2 will cover the surrounding workflow.

What's in a model?

At its core, the SDV is a set of machine learning models designed to understand and mimic real world data. Once the SDV creates a particular model, developers can generate synthetic data by sampling it. For synthetic data to be successful, this generative model must be correct — but through discussions with our open source community, we realized that there is no such thing as a single, winning approach that works every time. Each dataset and use case is different.

Our solution is to provide choices, giving users all the necessary tools to make useful synthetic data for each new case at hand. Let's dive into three popular uses of the SDV where such options are available: Tabular models, sequential data and business logic.

More Options for Tabular Models

The earliest version of SDV was based on a classic statistical method: Gaussian Copulas. This model is transparent by definition. It allows us to understand and exert control over formulas in the model, notably the distributions of each variable. This can be especially useful for business applications, where data often follows predictable distributions. For example, wind speed is known to follow a Weibull distribution, biological measures like height usually follow normal distributions and credit default rates often follow exponential distributions.

Meanwhile, advances in the AI space had also produced a robust, alternative model for those willing to sacrifice transparency: A deep learning technique called Generative Adversarial Networks (GANs). GANs model complex processes that don't follow known formulas. While these models’ inner workings aren’t easily explained by humans, they produce highly accurate results. We created a GAN, called CTGAN, specifically for synthetic data. This black box model is especially good at figuring out complex correlations between variables in large datasets.

For a long time, SDV allowed users a choice between our Gaussian Copulas based model, called GaussianCopula, and CTGAN to model tabular data. While this choice provided some flexibility, our users reported they had a hard time choosing between such extreme alternatives. We wondered if a middle ground was possible: Could we specify distributions while also using GANs to identify complex correlations?

We couldn't find any model that fit both of these requirements, so we made our own! A key insight was that we could use Gaussian Copulas to understand the data and transform it before applying it to a GAN. The result is CopulaGAN, a hybrid model we released in October 2020.

CopulaGAN combines the human accessibility of Gaussian Copulas with the robust accuracy of GANs. This innovation provides users with a new choice: a hybrid approach.

The Special Case of Sequential Data

Another tricky case pointed out by our users involved sequential data. While sequential data is stored in a table, it is unlike a regular table in that its rows are linked together, usually by a time component. This use case is extremely frequent, especially in finance — any table with credit card transactions, stock prices, or payments is almost certainly sequential.

At the time, solutions treated sequential data as a case of general tabular modeling. After all, sequential data is inside a table. However, these solutions failed to incorporate the key information that makes sequential data unique: The relationships that exist between rows.

While considering this pain point, we recognized sequential data as an entirely new case that required its own unique set of modeling techniques. In October 2020, we released our DeepEcho library, which focuses entirely on sequential data. We also introduced our PAR model: a GAN approach made specifically for sequential data.

Encoding Business Logic using Constraints

Even with a plethora of modeling choices, it's vital to capture nuances in business logic while modeling synthetic data. This is due to differences in how humans and machines understand datasets.

Often, humans can easily glean the meaning of a dataset using context clues. Consider a table showing the names and ages of students and their legal guardians. A human will intuitively realize that a student must be younger than their guardian.

But will a machine understand the same rule? Because all of the SDV's models use statistics, they analyze trends generally — meaning that in this case, they will include a small possibility that a student could be older than their guardian. After all, is it totally out of the question that an older individual could enroll and list their child as their guardian? Either way, only a human expert can truly figure out what makes sense for this dataset!

To solve this pain point, SDV introduced the concept of constraints in July 2020. Constraints give users the ability to encode their business knowledge and expertise into an SDV model. In our example, they could specify that a guardian's age must be greater than the student's. Currently, the GreaterThan and UniqueCombination constraints allow for easy handling of common scenarios. We also provide a blanket CustomConstraint class, which gives users flexibility to capture more nuanced knowledge.

More Community Feedback

We believe that the more humans and machines can work together, the more efficient our processes can become. In this article, we explained how user feedback about the SDV led to new core modeling techniques and innovations — enabling a system that now provides a choice of multiple models, handles sequential data, and understands constraints. In Part 2, we will discuss similar feedback-driven innovations in the rest of the workflow.

Using SDV — and giving us feedback — fuels this rapid evolution. To start a discussion, please message us on Slack or file an issue on GitHub. Working together, we can make SDV the most trusted, transparent and comprehensive platform for synthetic data generation!

For other inquiries, please contact info@sdv.dev.

Hello world! We, the creators of MIT's Synthetic Data Vault, warmly welcome you to our official blog. Soon we'll be using this space to deep-dive into topics related to our libraries, and to unpack ideas in the synthetic data space. We're looking forward to exploring this exciting area with you.

But first, we want to properly introduce our project: The Synthetic Data Vault (SDV), an open source software ecosystem for generating synthetic data. In this post, we’ll explain why synthetic data is important, and tell the story of how we created the vault. We’ll also lay out what’s in store — and how you can get involved. Let’s get started with a brief overview.

Synthetic Data What?

Synthetic data is a bold new frontier in machine learning. It allows developers to share and use data more effectively.

It may seem counterintuitive, but although billions of gigabytes of data are produced every day, there are still huge gaps in what developers are actually able to use. Accessibility concerns, regulatory issues and imbalanced datasets can all keep experts from using data. This impedes progress in finance, health care and other domains.

Good synthetic data can fill these gaps. The SDV uses machine learning to analyze data. Then, it creates fully synthetic datasets that mimic the original. Although the synthetic data is entirely machine generated, it maintains the original format and mathematical properties. This makes synthetic data versatile. It can completely replace the existing data in a workflow, or it can supplement the data to enhance its utility. Already, our users have successfully used the SDV to augment datasets, test applications, remove bias and more.

A History of the SDV

Our story starts in 2013. In MIT's Laboratory for Information and Decision Systems (LIDS), we were working on general data science projects. We had developed new techniques, and we were excited to test them on real datasets. However, as soon as we asked for the data, we hit roadblocks. The process for getting access to data turned out to be much more complex than we anticipated, with many regulations and security red tape.

We wondered: What if we didn't need the real data in the first place? If we had synthetic data with the same mathematical properties as the original, it would be much easier for everyone to share and use.

In 2016, we released a paper describing the very first iteration of the SDV. It introduced a novel technique for synthesizing multi-table data, and included trials where data scientists successfully used synthetic data instead of real data for machine learning tasks. Related research to come out of the lab included CTGAN, a novel approach to generating synthetic data using deep learning.

After these successes in the research community, we decided to move beyond purely academic solutions. Synthetic data has the potential to solve real-world problems faced by people on all sides of data science: internal developers writing software, external contractors working offshore, 3rd party partners offering services and even the end users who create the data. After some pilot testing on enterprise applications, we open sourced our work in 2018, publishing sdv on PyPi for general use. Open sourcing offered ample opportunities for collaboration and customization. It allowed users all over the world to test the SDV in enterprise settings, and helped the SDV ecosystem evolve into a one-stop shop for synthetic data needs!

We listened to feedback and, as of today, have made 93 releases (across all our libraries), addressing 504 issues. We have been thrilled to see a burgeoning community of invested users using the SDV to solve problems. We've seen over 200K user downloads from PyPi, 400 stars in the SDV GitHub repository and 200 developers in our Slack channel. Our community is global and includes people in diverse roles: academics, data scientists, operations managers, engineers and more. We are continually learning from our community, and we're excited to bring new innovations to you!

Just the Beginning

Synthetic data has the potential to revolutionize the entire field of data science, allowing us to solve problems that once seemed untouchable. We want the Synthetic Data Vault to be the most trusted, transparent and comprehensive platform for synthetic data generation, but we can't do it without our users. It's our ever-growing open source community that allows us to quickly repair bugs, triage feature requests and improve to serve a variety of real-world needs.  

That’s where you come in. If you’re already a member of this community, we can’t thank you enough. And if you’d like to get involved, see below for ways to get started. Either way, watch this space for more nuanced discussions about synthetic data. We're excited to share what we've learned from you, and show how we are collectively improving the ecosystem. It’s time to open the vault!

Want more ways to get involved?

• Follow us on Twitter @sdv_dev for release announcements, blog updates and more
• Join our Slack community to meet other users, discuss synthetic data solutions and suggest topics for the blog
• Visit & star our GitHub repositories
• If you've successfully used the SDV for your project, share your experience and tag us

For other inquiries, please contact us at info@sdv.dev.

*Total download statistics per continent come from the Linehaul project using BigQuery, and include mirrors. Are you aware of more accurate ways to count Python package downloads? Let us know!