In a previous article, we discussed why, when developing our open source libraries, we emphasize growing our overall users – not just our contributors. We elaborated on why we focus on everyone who uses our software to solve a problem, as opposed to following the more traditional open source practice of catering specifically to code contributors.

But we cannot stop at defining our focus – we have to put it into practice. In this article, we'll share some practical strategies we have learned in the course of adopting this more user-centric mindset.

We have come to these strategies by regularly interacting with the users who have adopted the SDV ecosystem, and we've iterated them until we found success. (Some of our libraries are already at 1 million downloads!) This has given us confidence in our approach, which we're excited to share below.

1. Find the right channels to reach more users

Our first strategy involves our overall presence as an open source ecosystem.

Shifting our focus to users has allowed us to think more critically about who we are trying to help, and whether we are actually reaching them. For example, we were initially prioritizing checking and responding to questions on GitHub, a platform that makes it easy to reference technical material and scrutinize bugs.

But GitHub caters primarily to contributors – and thus leaves out the rest of the users we'd like to reach. In fact, those users may not have GitHub accounts at all! They are more likely to feel at home when they can ask us questions directly, working with us to improve their understanding. They often don't have the time to dig through technical discussions, or the desire to create an issue on GitHub, especially if their question is more fundamental. (Since “issue creation” has always been defined as a part of the software development lifecycle, users may not feel comfortable asking more basic questions there.)

To find the users we wanted, we decided to expand our presence to other platforms that cater to their needs. We have found Slack to be a great solution – it is welcoming and easy to use, and enables direct communication. Today, our Slack is a fast-growing community of over 800 members, and a new space for us to learn about how people use our software.

2. Users are just as important as contributors

Our attitude towards users matters just as much as their ability to find us.

As the core maintainers for an open source project, we all have a deep passion for software. It is natural for us to want our users to share this passion – and also natural to perceive a lack of initiative if a user hasn't understood certain concepts. But in our drive to recognize the importance of all users, we have learned to understand — and even embrace — that many users have different needs and time pressures than we are used to.

For example, we frequently receive questions about how to upload a CSV file into Python. This is a standard data science procedure, so some might label this question as "lazy." We don't believe that's true. In fact, these users may be picking up Python for the first time because they think our software could solve their problem, which shows a lot of initiative. They are not unqualified to use our library; they might just need a helping hand.

To figure out what will actually help users, we put ourselves in their shoes. This mindset has led to some of our current best practices:

• Empathize with the user's pain points. Working on software openly means that we'll get more feedback more often. Often, an issue identified by a user may already be on our roadmap, is inspiring internal debates, or is on hold until we have more resources. When we're reminded of such an issue, it can be easy to get defensive and engage in a debate – which ultimately wastes everyone's time. Instead, we use the opportunity to build camaraderie. We always try to replicate users' issues, which helps us acknowledge the frustration because we feel it too. No software is perfect!
• Focus on the problem, not the solution. Because our users do not have our domain expertise, they'll sometimes request features that seem difficult to accommodate. When this happens, we remind ourselves that it's not the user's job to understand our system. Rather, it's our duty to dig deeper and find the root of the concern. This helps us design new features in a way that matches our vision and satisfies users.
• Above all else, move them forward. When a user has a request, we aim to provide a timely, focused response so that they can take the next step in their usage journey. If we can't immediately resolve the issue, we provide workarounds that allow their projects to proceed. This is more difficult than it seems. At times, we want to passionately respond with our own long-term vision — but this is not useful to users who just want their project to work.

The illustration below shows a hypothetical example of using these best practices.

These best practices reflect our overall attitude, which elevates users to the same level of importance as open source contributors.

3. Go the extra mile – it only takes a few minutes!

Going above and beyond can mean creating special material for users and learning to speak their language. By now, it has become standard practice for us to disambiguate and translate our technical communications for a more general audience.

Our SDV ecosystem is filled with examples of conveying the same information in multiple ways. Below are some excerpts from our announcement of a new version of SDV (0.16.0) in July 2022.

The SDV open source contributors are familiar with technical concepts like “unify sampling params for reject sampling” or “Add create_custom_constraint factory method”. They're also interested in following along with specific GitHub issues, which link to the code changes.

Meanwhile, user-centric communication focuses on the pain points that we've solved. This is informative for current users and welcoming for new ones. As a result, users coming to our library for the first time can scan through the Slack channel to see what we're working on. Best of all, because we're thinking in these different ways already, it only takes a few minutes to draft these different types of announcements!

Conclusion

Adopting a user-centric mindset has significantly contributed to our open source growth. We started by identifying users and finding the right channels to reach them, which naturally expanded our open source presence. Then we learned to empathize with users and embrace their needs, which has manifested as more productive conversations and relationships. Finally, we always think it's great to go above and beyond – especially if it only takes a few minutes!

Are there any strategies we've missed? Let us know what you think in the comments below!

Defining an Open Source User for 2023 and beyond

Code contributors are an essential part of an open source (OS) project. But in our experience, making code contributors the sole focus of an open source project ends up disenfranchising another large segment of important people: a library's users. This segment, we have found, is more critical to our success, providing indispensable feedback, finding use cases and helping us to improve our open source and the product that relies on it. In this first article (in a series), we synthesize key attributes that we use to identify a user.

Traditionally, open source libraries have been centered around software development, as collaborating on code is vital for maintaining complex software. It has become customary to use the number of unique code contributors as a core metric of a given library's success. Metrics of success drive how the overall ecosystem is maintained, including how software is designed (APIs), the audience for which usage guides are developed, the types of demos that are built, and how communications are handled. To bring users into fold, all these need to be revisited keeping them in mind.

At the same time, open source is proving to be a successful model for startups. As the core maintainers of the Synthetic Data Vault project — the world's largest open source library for modeling and generating tabular synthetic data — we are constantly striving to realize the benefits of this model firsthand. For us, open source has been vital to building a trusted and usable machine learning system. With this and subsequent articles we are synthesizing our current thinking about open source, and some key lessons we have learned on our way to this point.

Who is a user?

Our definition of a user is: Anyone who attempts to use our open source library to solve their problem. Generally, users:

• …are goal-oriented. A user comes to our library with a specific project that they're working on.
• …have limited time. A user often has a deadline for their project. They may not have time to learn the nitty-gritty details of our software, or engage in deeper conversations about its development.
• … have different expertise. A user is probably coming to our library to help with a project in their own domain, whether that's healthcare, clean energy or something else. They might not have the same knowledge base as a professional software developer would (although they also might — more on this later).

While this definition may seem straightforward, these attributes have become the cornerstone of how we maintain and communicate about our software, and  how we develop APIs. They have also inspired the main question we use to measure our progress: Is our library making a material difference in users' projects? In subsequent articles, we plan to share how we applied these strategies to build the largest open source user community around synthetic data, and what we have learned in the process.

What changes are we making to set up an OS for success?

Charting this path with a laser-sharp focus on the “user” has required us to address some commonly asked questions up front, both for our team and externally. Here are just a few.

Users are developers too and probably more critical for our success

Just because a user isn't interested in learning the internal details of our software doesn't mean we can automatically categorize them as not a developer. They may be experts in other fields and may be developing software there. In addition, they are still using our Python API to help them with their project — and therefore, they are developing software. To expand and serve our user base, we focus our efforts on what we want to achieve with our open source strategy, rather than creating different strategies based on the perceived skill level of who is using our library. As a result, we want every API we publish to be understandable and usable by everyone. We want our communications to be cognizant of the fact - they don't have time! In 2023, we believe that everyone is a developer — or at least, we like to serve everyone and make them part of the software movement.

User friendly APIs are game changers

One question we asked ourselves was "shouldn't the user friendliness delegated to graphical user interfaces (GUIs)?". GUIs finalize a straight, stepwise process to successful project completion, while the code provides flexibility to try things slightly differently. When they feel restricted by the straight stepwise process for their specific project/use case, pioneering users instead use code. Creating a user-friendly API that lets users apply our open source to their project in a transparent way, and provides access to different metrics and progress states at different stages, gives our users a great chance at succeeding. It also helps us to efficiently discover more pathways, and most importantly, more use cases. This makes our open source essentially a low-code version of what goes into the product.

Github stars are not enough

Github star histories are regularly used to indicate an exponential growth curve for a library. They are often considered a leading indicator for a need in the market that the library may be targeting, or top of the funnel for an open source, and there are now well-developed strategies for growing stars over time. Used effectively, we find these strategies to be a good marketing tool, and well-intentioned for increasing the top of the funnel. We ourselves use them from time to time, as they increase reach and can bring in more users. But we find that they should be balanced with feature development, carefully listening to users, and measuring how often folks are downloading and using the library and raising issues. Star growth should be followed by growth in downloads and issues raised by users.

We look forward to discussing our experiences with open sourcing in 2023 and beyond. In the articles that follow, we will share some more of our strategies and measures for engagement. We welcome any thoughts, comments, suggestions and questions below.

A dataset can unlock many doors for your organization, helping with everything from predictive forecasting to data-driven decision making. But in some situations, you may not have all of the data that you need. A common scenario is having too few data points, which can lead to an imbalance of variables.

In this article, we'll take a closer look at this scenario. We'll recap why this can be a problem for your projects and walk you through some possible solutions. We'll end by explaining why synthetic data might be especially useful for overcoming this challenge.

Why is it a problem to have a data imbalance?

At a basic level, data is a record of events, and a data imbalance happens when some events occur much less frequently than others. For example:

• In healthcare, cancer occurs less frequently than diabetes
• In finance, it's rare to see fraudulent credit card charges
• In local government, it's rare to have a day with a major natural disaster such as a fire

A natural data imbalance isn't inherently a problem, but it can become one if the rare events are important. As an example, let's assume you're working at a hospital that is treating COVID patients. One day a new COVID variant – let's call it Variant X – appears in the population. Since it is so new, it currently occurs very rarely (<2.5% of the time). This leads to a data imbalance for this COVID variant, as illustrated below.

This imbalance is a problem when it's critical to account for Variant X. For example, you may want to build a predictive model for who is most likely to be hospitalized. If you use the data as-is, your model may only consider Omicron (the majority) and treat Variant X as an outlier. This can lead to poor predictions – and bad planning – because Variant X may soon become the dominant strain.

Using Data Augmentation to Fix Imbalances

In an ideal world, your data would include more patients with Variant X. But until then, you need to find a solution that will allow you to produce reasonable predictions. What if you create some artificial patient data for the sake of making a robust predictive model?

Let's assume you have data for 1,000 COVID hospital patients, 975 with the Omicron variant and 25 with Variant X. If you can create 950 additional artificial Variant X patients, then you can create an evenly-balanced dataset. This process is illustrated below.

You may be skeptical because there are only 25 patients with Variant X to begin with. How can we reasonably produce 950 more based on that? As usual with data science, the devil is in the details. Let's go through some approaches to see what works.

Attempt #1: Oversampling

Your first instinct may be to take the existing 25 Variant X patients and weigh them more heavily. One easy way to achieve this: You can duplicate each original patient 40 times to get 1,000 patients.

In data science, this is known as oversampling. Sometimes, this is done programmatically, sampling patients (with replacement) as many times as needed. Other times, this can be achieved using mathematical formulas to provide weights. An illustration is shown below.

With oversampling, Variant X is no longer rare, so your model cannot ignore it. But if you actually use this data, your project may not be successful. Your model may confidently predict that all Variant X patients must be over the age of 50. But this is not necessarily right – just because the existing patients had these characteristics doesn't mean Variant X patients always will.

The mistake was over-emphasizing the same set of patients, making the model more likely to create strong claims. This is commonly referred to as overfitting the data: The model over-emphasizes the importance of a small number of records, and makes blanket predictions that lack nuance.

Attempt #2: Randomizing

To avoid the problems that come with oversampling, let's explore the opposite direction for argument's sake: What if we created artificial Variant X patients by choosing variables completely at random? An example is illustrated below.

Randomization avoids overfitting because no patient is repeated. But this approach introduces problems of its own: You may find that the data doesn't make sense anymore. The example data above highlights some problems that can arise. We see an artificial 23-year-old patient with dementia, and many diabetic patients with low BMIs. In the medical world, these events are not likely and indicate that there is a problem with the data.

These inconsistencies may (rightfully) dissuade you from using randomization. Since random data lacks patterns, a model will not be able to draw conclusions from it. In data science, we call this a problem of noisiness. Noisy data has too many random combinations to produce any useful learnings.

A Better Solution: Defining Neighborhoods

So far, we've seen attempts at extreme ends.

1. Oversampling will emphasize one set of patients, leading to an overfit model
2. Randomizing will make the dataset noisy, precluding useful conclusions

The solution we need falls somewhere in the middle: We'd like to loosely base the artificial patients on the real ones. This is related to the data science concept of neighborhoods. Drawing a neighborhood around some patients identifies general commonalities between them – without setting any strict rules. For example, Variant X patients may be more likely to have had a known exposure, but it's not guaranteed. Note that there is no exact definition for a neighborhood. It can change based on our assumptions and how broad we want to make it.

Once we know a neighborhood, we can create artificial patients that are inside it. These patients won't be exactly the same as the existing ones, but they won't have completely random values either.

Synthetic Data for Label Balancing

A compelling solution for discovering neighborhoods is synthetic data. A synthetic data software – such as our open source Synthetic Data Vault (SDV) – uses machine learning to learn patterns from real patients, and then creates synthetic patients.

The SDV discovers neighborhoods at a variety of levels in the form of trends. It's able to learn overall trends (for all patients) as well as trends that are unique to a variable (such as Variant X). For example:

• For all patients, a higher age corresponds to a greater risk of dementia and a higher BMI corresponds to a greater risk of diabetes.
• Variant X patients tend to be older, while Omicron patients tend to be younger.
• Etc.

As a result, synthetic patients have some variation – but the data still makes sense in context. An example table of SDV-generated patients is shown below.

This is the middle solution we were looking for: Synthetic data won't cause overfitting and is less noisy than randomization. The best part is that there are multiple synthetic data techniques and settings available in the SDV, providing flexibility and tradeoffs.

Takeaways

In this article, we explored imbalanced datasets. It is common to have an imbalanced dataset due to rare events – which becomes a problem if those rare events are important for your project.

Fixing the imbalance problem requires a careful tradeoff between overfitting data and creating noisy data. Synthetic data is a compelling solution that achieves a middle ground by discovering neighborhoods of similar data. This allows you to realistically fix the imbalance without resorting to either extreme.

Are you interested in label balancing? Have you already explored using the SDV for this problem? Drop us a comment below!

This article was researched by Santiago Gomez Paz, a DataCebo intern. Santiago is a Sophomore at BYU and an aspiring entrepreneur who spent his summer learning and experimenting with CTGAN.

The open source SDV library offers many options for creating synthetic data tables. Some of the library's models use tried-and-true methods from classical statistics, while others use newer innovations like deep learning. One of the newest and most popular models is CTGAN, which uses a type of neural network called a Generative Adversarial Network (GAN).

Generative models are a popular choice for creating all kinds of synthetic data – for example, you may have heard of OpenAI's DALL-E or ChatGPT tools, which use trained models to create synthetic images and text respectively. A large driver behind their popularity is that they work well — they create synthetic data that closely resembles the real deal. But this high quality often comes at a cost.

Generative models can be resource-intensive. It can take a lot of time to properly train one, and it's not always clear whether the model is improving much during the training process.

In this article, we'll unpack this complexity by performing experiments on CTGAN. We'll cover –

• A high-level explanation of how GANs work
• How to measure and interpret the progress of CTGAN
• How to confirm this progress with more interpretable, user-centric metrics

Since the library is open source, you can see and run the code yourself with this Colab Notebook.

How do GANs work?

Before we begin, it's important to understand how GANs work. At a high level, a GAN is an algorithm that makes two neural networks compete against each other (thus the label “Adversarial”). These neural networks are known as the generator and the discriminator, and they each have competing goals:

• The discriminator's goal is to tell real data apart from synthetic data
• The generator's goal is to create synthetic data that fools the discriminator

The setup is illustrated below.

This setup allows us to measure – and improve – both neural networks over many iterations by telling them what they got wrong. Each of these iterations is called an epoch, and CTGAN tracks inaccuracies as loss values. The neural networks are trying to minimize their loss values for every epoch.

The CTGAN algorithm calculates loss values using a specific formula that can be found in this discussion. The intuition behind it is shown below.

As shown by the table, lower loss values – even if they are negative – mean that the neural networks are doing well.

As the epochs progress, we expect both neural networks to improve at their respective goals – but each epoch is resource-intensive and takes time to run. A common request is to find a tradeoff between the improvement achieved and the resources used.

Measuring progress using CTGAN

The open source SDV library makes it easy to train a CTGAN model and inspect its progress. The code below shows the steps. We train CTGAN using a publicly available SDV demo dataset named RacketSports, which stores various measurements of the strokes that tennis and squash players make over the course of a game.

from sdv.demo import load_tabular_demo
from sdv.tabular import CTGAN

metadata, real_data = load_tabular_demo('RacketSports', metadata=True)
table_metadata = metadata.to_dict()

model = CTGAN(table_metadata, verbose=True, epochs=800)
model.fit(real_data)

As part of the fitting process, CTGAN trains the neural networks for multiple epochs. After each epoch, it prints out the count, the generator loss (G) and the discriminator loss (D). Keep in mind that lower numbers are better – even if they are negative. An example is shown below.

Epoch 1, Loss G:  1.0435,Loss D: -0.1401
Epoch 2, Loss G:  0.4489,Loss D: -0.1455
Epoch 3, Loss G:  0.4756,Loss D: -0.0956
Epoch 4, Loss G:  0.3902,Loss D:  0.0344
Epoch 5, Loss G:  0.0912,Loss D:  0.3030
...

To see how the neural networks are improving, we plot the loss values for every epoch. The results from our experiment are shown in the graph below.

Based on the characteristics of this graph, it's possible to deduce how the GAN is progressing.

Interpreting the loss values

The graph above may seem confusing at first glance: Why is the discriminator's loss value score oscillating at 0 if it is supposed to improve (minimize and become negative) over time? The key to interpreting the loss values is to remember that the neural networks are adversaries. As one improves, the other must also improve just to keep its score consistent. Here are three scenarios that we frequently see:

1. Generator loss is slightly positive while discriminator loss is 0. This means that the generator is producing poor quality synthetic data while the discriminator is blindly guessing what is real vs. synthetic. This is a common starting point, where neither neural network has optimized for its goal.
2. Generator loss is becoming negative while the discriminator loss remains at 0. This means that the generator is producing better and better synthetic data. The discriminator is improving too, but because the synthetic data quality has increased, it is still unable to clearly differentiate real vs. synthetic data.
3. Generator loss has stabilized at a negative value while the discriminator loss remains at 0. This means that the generator has optimized, creating synthetic data that looks so real, the discriminator cannot tell it apart.

It is encouraging to see that the general pattern for the RacketSports dataset is similar to a variety of other datasets. These are shown below.

Of course, other patterns may be possible for different datasets. But if loss values are not stabilizing, watch out! This would indicate that the neural networks were not able to effectively learn patterns in the real data.

Metrics-Powered Analysis

You may be wondering whether to trust the loss values. Do they indicate a meaningful difference in synthetic data quality? To answer this question, it's helpful to create synthetic data sets after training the model for different numbers of epochs, and assess the quality of the data sets.

NUM_SYNTHETIC_ROWS = len(real_data)

synthetic_data = model.sample(num_rows=NUM_SYNTHETIC_ROWS)

It is important to select a few key metrics for a quantifiable quality measure. For our experiments, we chose 4 metrics from the open source SDMetrics library:

• KSComplement evaluates the shape of numerical columns
• TVComplement evaluates the shape of discrete columns
• CorrelationSimilarity evaluates pairwise correlations between columns
• CategoryCoverage evaluates whether the synthetic data covers all possible values

Each metric produces a score ranging from 0 (worst quality) to 1 (best quality). In the example below, we use the KSComplement metric on a numerical column in the RacketSports dataset.

from sdmetrics.single_column import KSComplement

NUMERICAL_COLUMN_NAME='dim_2'

score = KSComplement.compute(
   real_data[NUMERICAL_COLUMN_NAME],
   synthetic_data[NUMERICAL_COLUMN_NAME])

Our results validate that the scores do, indeed, correlate with the loss value from the generator: The quality improves as the loss is minimized. Some of the metrics – such as CorrelationSimilarity and CategoricalCoverage – are high to begin with, so there is not much room to improve. But other metrics, like KSComplement, show significant improvement. This is shown in the graph below.

It's also possible to visualize the synthetic data that corresponds to a specific metric. For example, KSComplement compares the overall shape of a real and a synthetic data column, so we can visualize it using histograms.

from sdmetrics.reports import utils

utils.get_column_plot(
  real_data,
  synthetic_data,
  column_name=NUMERICAL_COLUMN_NAME,
  metadata=table_metadata)

Overall, we can conclude that the generator and discriminator losses correspond to the quality metrics that we measured – which means we can trust the loss values, as well as the synthetic data that our CTGAN created!

Conclusion

In this article, we explored the improvements that the CTGAN model makes as it iterates over many epochs. We started by interpreting the loss values that each of the neural networks – the generator and the discriminator – reports over time. This helped us reason about how they were progressing. But to fully trust the progress of our model, we then turned to the SDMetrics library, which provides metrics that are easier to interpret. Using this library, we could verify whether the reported loss values truly resulted in synthetic data quality improvements.

This may lead us to a new, potential feature: What if we integrated these easily interpretable, user-centric metrics into the CTGAN training progress? This feature would allow you to specify the exact metrics you'd like to optimize upfront – for example, KSComplement. In addition to the generator and discriminator loss, CTGAN may be able to report a snapshot of this metric. A hypothetical example is shown below.

model = CTGAN(
  table_metadata,
  verbose=True,
  epochs=800,
  optimization_metric='KSComplement',
  optimization_column='dim_2')
  
model.fit(real_data)

Epoch 1, Loss G: 1.0435, Loss D: -0.1401, KSComplement: 0.7832
Epoch 2, Loss G: 0.4489, Loss D: -0.1455, KSComplement: 0.7671
Epoch 3, Loss G: 0.4756, Loss D: -0.0956, KSComplement: 0.7664
…
Epoch 200: Loss G: -2.542, Loss D: 0.0002911, KSComplement: 0.92391

Such a feature would allow more transparency over CTGAN's learning process, and allow you to stop training your models once the metrics are high.

What do you think? If you're interested in exploring the inner workings of CTGAN and optimizing your synthetic data, drop us a comment below!

In recent years, synthetic data has shown great promise for solving a variety of problems – like addressing data scarcity for AI and overcoming barriers to data access. As your organization becomes serious about adopting synthetic data, it's crucial to incorporate the right metrics and evaluation frameworks into your projects.

Since the synthetic data space is so new, there aren't yet industry standards for setting and measuring outcomes. At DataCebo, we've worked with a variety of teams using synthetic data. In this article, we're sharing the best practices we've learned along the way, as well as the one key mistake to avoid.

What are synthetic data metrics?

In some fields – such as synthetic image generation – it's easy to visually inspect the output (in this case, synthetic images) and determine its quality. But if you are creating synthetic data in a tabular format (with rows and columns), it's difficult to make an overall assessment just by looking at the raw data. This is evident in the table below:

For tabular synthetic data, it's necessary to create metrics that quantify how the synthetic data compares to the real data. Each metric measures a particular aspect of the data – such as coverage or correlation – allowing you to identify which specific elements have been preserved or forgotten during the synthetic data process.

In our open source library, SDMetrics, we've provided a variety of metrics for evaluating synthetic data against the real data. For instance, you can use the CategoryCoverage and RangeCoverage metrics to quantify whether your synthetic data covers the same range of possible values as the real data:

You may also be curious about whether the synthetic data captures trends between pairs of columns. To compare correlations, you can use the CorrelationSimilarity metric:

The SDMetrics library has over 30 metrics, with more still in development.

But having access to metrics is just one part of the story. With so many metrics, it can be difficult to decide which ones to focus on – and how to make progress in your synthetic data project. To successfully deploy synthetic data, it's important to consider metrics during all steps of your project development cycle.

In the rest of this article, we'll share a 3-step plan for incorporating metrics – and the SDMetrics library – into your synthetic data project to increase your chances of success.

Step 1: Start with the project goals

It is tempting to create synthetic data quickly and then test it using all the available metrics. After all, it's hard not to be curious about what synthetic data can do! But to succeed with your project, it's important to take a step back and focus on the problem you are trying to solve first.

Synthetic data creation isn't an end in itself. Just as with most data work, you don't create synthetic data for its own sake — you use it to solve a problem. If you want your synthetic data project to succeed, pay close attention to what that problem is, as this will help you narrow down a few key metrics.

For example, imagine that your organization has two different synthetic data projects related to software testing and machine learning, respectively. Because these projects have different goals, you’ll need to consider different metrics:

Focusing on your goals allows you to identify which metrics are important for the ultimate success of your project. The single biggest mistake we see people making is to skip this critical step. Without focus, you can easily get bogged down running multiple tests and tweaking your synthetic dataset, rather than meeting the specific considerations for your project. This can derail your efforts – leading to a complex project that doesn't add any value.

Step 2: Let your goals guide the synthetic data creation

Your goals can help you appropriately scope your project and cut costs. A core subset of metrics can guide your synthetic data creation, making it faster and more targeted to your needs.

Chances are, you'll be faced with many decisions throughout your project. For example, in the SDV library, there are 5 different algorithms that create synthetic data, each with their own settings that lead to hundreds of potential models you can create. But if you know that your synthetic data project is software testing, you've identified that coverage, boundaries and business rules are the highest priority metrics. This will guide your decision-making.

In this case, you may find success choosing our preset model, FAST ML. This model uses statistical methods to achieve your minimal requirements while also providing high performance – FAST ML can train a mid-size data table (100 columns and 100K rows) in only a few minutes. You can compare this to other GAN-based models that are more resource-intensive, taking hours to finish. If your project metrics are satisfied with FAST ML, it is reasonable to choose this model over a GAN, even if it isn't perfectly optimized across all possible metrics.

From this example, we can see that metrics are not just something to evaluate at the end of the project – they are useful tools for decision-making throughout your project.

Step 3: Test the end-to-end workflow upfront

The purpose of metrics is to provide guardrails and focus for your project – scoping it so that you can drive business value most efficiently. For the highest chances of success, it's important to apply the synthetic data end-to-end for downstream applications, so that you can verify that business value upfront.

Continuing with our example of software testing, it's important to use the synthetic data for your downstream software testing suite as quickly as you can to verify the benefits of synthetic data. If you've chosen your metrics correctly and considered them when making decisions (steps #1 and #2), then you'll see that this translates to business value.

This type of cost-benefit analysis can help you make the case for synthetic data adoption within your enterprise. It can also help you iterate – to get more business value from your synthetic data, you can continue to optimize the metrics you've chosen in step #1, or identify new ones.

The Takeaway

Technically, there are an infinite number of metrics that could be used to evaluate synthetic data. The key to success is to incorporate select metrics into your synthetic data project development rather than just applying all metrics at the end.

Your project goals are critical to helping you choose the right metrics. Setting them upfront allows you to make better decisions during your project development. And going end-to-end allows you to measure the business value that your synthetic data brings to the organization.

What are your thoughts? Leave comments below! If you noticed other evaluation pitfalls in your projects, let us know below or reach us directly at info@sdv.dev.

This article was researched by Arnav Modi, a community user. Arnav is a high school student and aspiring data scientist who spent his summer learning about the SDV and how synthetic data is used to perform ML tasks.

One potential use for synthetic data is to replace real data in the development of new machine learning (ML) models. Imagine a scenario where you need to build a predictive ML model – perhaps for a function critical to your business, like predicting customer satisfaction or sales success – with one important consideration: The data is sensitive, so only trusted employees can access it with specific credentials.

Access to sensitive data may create a barrier for a variety of reasons:

• You might not have ML expertise in your organization, which means you need to use external software or contractors to complete the task. However, you are unable to share the data with them.
• Your data is available on a secure, cloud-based platform for trusted employees to access remotely. They work on this data using interactive notebooks. Every time they lose their connection – due to WiFi outages, their laptops falling asleep, etc. – they may lose their work or have to reconnect.
• You have a robust authentication system that your team uses. However, it creates a barrier to entry for rapid, iterative collaboration between members, sharing work and debugging data pipelines. As a result, your collaboration is much slower than it would be if your team could access the data without the need to authenticate.

In cases like this, synthetic data can be an ideal solution: You can create synthetic data based on the original, sensitive data set, and use it more freely during ML development.

One key question will determine if this method succeeds: Is the synthetic data actually useful for your ML task? We performed an experiment to find out.

In the rest of this article, we'll describe our experimental setup and findings. (You can double-check our work in this Colab Notebook.)

Experimental Setup

If an ML model is trained using synthetic data instead of real data, what happens to the model's performance? To answer this question, we identified 3 publicly available datasets (Income, Bank and Airline) that are associated with particular ML prediction tasks. The datasets and tasks are summarized below.

Our experiment compared the performance of an  ML model trained on the original data, vs. one trained on the synthetic data provided by the SDV.

• Control (Original data): How successfully can we complete the ML prediction task if we use the real data? Because some predictions are harder than others, this control helped us identify the overall difficulty of these specific tasks.
• Experiment (Synthetic data): How successfully can we complete the ML prediction task if we use synthetic data instead? We used the SDV's CopulaGAN to generate synthetic data from the three original datasets.

In order to develop and test the ML model, we turned to the SDMetrics library — specifically the ML Efficacy metrics, which build an ML model and evaluate its performance. We used the Binary Decision Tree Classifier and Binary Logistic Regression models. The overall experimental setup is illustrated below.

To obtain reliable findings, we ran 3 iterations and averaged the results.

Results

The graph below shows how well we are able to perform an ML task using the original vs the synthetic data.

Discussion

The original data quantifies the general difficulty of the ML task. Looking at these values, we can see that the Income Dataset is the hardest task, as neither of our methods were able to get above 90% accuracy using the original data.

Comparing the datasets allows us to quantify the suitability of synthetic data for ML development. Our results show a loss of between 1 and 9% of the original efficacy value for all comparisons, with a median loss of roughly 2.5%.

It's important to note that the simplifications we've made for this experiment may be resulting in worse accuracy than we would see in real-world use.

• Applying CopulaGAN out-of-the-box to each dataset is simplistic. In a real-world scenario, the model's parameters would likely be explicitly tuned and constraints would be used to improve synthetic data quality.
• The Decision Tree and Logistic Regression evaluators are relatively simplistic ML classifiers. An ML expert (or ML software) might use more advanced techniques.
• In our scenario, the 3rd party delivers a fully trained, ready-to-go ML model. Another approach is to ask them to use the synthetic data to deliver an untrained model – so that you can train it yourself on the real dataset. This alternative setup, which should increase the prediction accuracy, will be a topic for a future article.

In summary, the accuracy loss we observe represents the worst case scenario. In a production environment, higher-quality ML models and more careful tuning of the SDV will likely minimize performance differences between original and synthetic data.

Takeaways

In this article, we quantified the effect of replacing real data with a synthetic data clone for ML development. Our results show a loss of 2.5% accuracy when using synthetic data. Considering these results, we assess that it is reasonable to explore the use of synthetic data for the purpose of ML development.

In order to maximize the utility of the synthetic data, we recommend tuning the SDV model and using constraints to improve the data quality. In future articles, we'll explore more details about using synthetic data for ML.

Are you using the SDV to solve your ML business needs? Publish your findings on the SDV blog as a guest author! Contact us at info@sdv.dev.

By default, a machine learning model (ML) may not always learn the deterministic rules in your dataset. We've previously explored how the SDV allows user to input their logic using constraints. With constraints, an SDV model produces logically correct data 100% of the time.

While an end user might expect the constraint to "just work," engineering this functionality requires some creative techniques. In this article, we'll describe the techniques we used to build the UniqueCombinations constraint. You can also follow along in our notebook.

!pip install sdv==0.13.1

import numpy as np
import warnings

warnings.filterwarnings('ignore')

What is a Unique Combinations Constraint?

Users frequently encounter logical constraints on the permutations -- mixing & matching -- that are allowed in synthetic data.

To illustrate this, let's use the world_v1 dataset from the SDV tabular dataset demos. This simple dataset describes the population of different cities around the world.

from sdv.demo import load_tabular_demo

data = load_tabular_demo('world_v1')
data = data.drop(['add_numerical'], axis=1) # not needed for this demo
data.head()

Relationship between Name, CountryCode and District

Looking at the data, we can observe that there is a special relationship between the Name of the city, its CountryCode and its geographical District: When generating synthetic data, the model should not blindly mix-and-match these values. Instead, it should reference the real data to verify whether the combination is valid. This is called a UniqueCombinations constraint.

For example, take a particular city, like Cambridge, which appears 3 times in our dataset.

data[data.Name == 'Cambridge']

The constraint states that Cambridge should only ever appear with GBR (England), CAN (Ontario) or USA (Massachusetts). It is invalid if it appears in any other region -- for eg. Cambridge, France.

How does the SDV handle a Unique Combination out-of-the-box?

Let's try running the sdv as-is on the dataset to see what happens. We'll use the GaussianCopula model on our dataset.

from sdv.tabular import GaussianCopula

np.random.seed(0)

model = GaussianCopula(
  categorical_transformer='label_encoding' # optimize speed
) 
model.fit(data)

Now, let's generate some rows to inspect the synthetic data.

np.random.seed(12)
model.sample(5)

Although the sdv is generating known city names, countries and districts, their combinations don't make sense. We can also go back to our original example and generate only some rows for Cambridge.

np.random.seed(10)

conditions = {'Name': 'Cambridge'}
model.sample(5, conditions=conditions)

The result is a variety of Cambridges that aren't necessarily in USA, GBR, or CAN. These aren't valid cities!

What's going on? The SDV models include probabilities that some unseen combinations are possible. This is by design: Synthesizing new combinations -- that don't blatantly match the original data -- helps with privacy.

However in this particular case, we aren't worried about the privacy of a city belonging to a country or district. We actually do want the data to match. This is why we need to build a constraint.

Fixing the data using rejecting sampling

In our previous article, we described a solution called reject_sampling that works on any type of constraint and is very easy to build: We simply create the synthetic data as usual and then throw out (reject) any data that doesn't match.

In theory, this can solve our UniqueCombinations constraint. In practice, this strategy is only efficient if the model can easily generate acceptable data. Let's calculate the chances of getting an acceptable combination (Name, CountryCode, District) from the model.

np.random.seed(0)

# Sample data from the model
# The sample may include combinations that aren't valid
n = 100000
new_data = model.sample(n)

# Calculate how many rows are valid
combo = ['Name', 'CountryCode', 'District']
merged = new_data.merge(data, left_on=combo, right_on=combo, how='left')
passed = merged[merged['ID_y'].notna()].shape[0]

# Print out our results
print("Valid rows: ", (passed/n)*100, "%")
print("Rejected rows: ", (1 - passed/n)*100, "%")

Valid rows:  0.038 %
Rejected rows:  99.96199999999999 %

With such a low probability of passing the constraint, this strategy can become intractable.

Fixing the data using transformations

A more efficient strategy is for the ML model to learn the constraint directly, so it always produces acceptable data. We can do this by transforming the data in a clever way, forcing the model to learn the logic.

Our previous article described how to do this for a different constraint. Unfortunately, the exact same transformation won't work to solve our current UniqueCombinations constraint. The transform strategy requires a different, creative solution for each constraint. So we have to start from scratch.

Can you think of any other ways to enforce UniqueCombinations?

A solution: Concatenating the data

One solution is to concatenate the data. That is, rather than treating the city Name, CountryCode and District as separate items, we treat them as a single value. This will force the model to learn them as 1 single concept rather than as multiple columns that can be recombined.

Let's see this in action.

# create transformed data that concatenates the columns
data_transform = data.copy()

# Concatenate the data using a separator
data_transform['concatenated'] = data_transform['Name'] + '#' + data_transform['CountryCode'] + '#' + data_transform['District']

# We can drop the individual columns
data_transform.drop(labels=['Name', 'CountryCode', 'District'],
                    axis=1, inplace=True)

data_transform.head()

Now, we can train the model using the transformed (concatenated) data instead.

np.random.seed(35)

# create a new model that will learn from the transformed data
model_transform = GaussianCopula(categorical_transformer='label_encoding')
model_transform.fit(data_transform)

# this will produce transformed data
output = model_transform.sample()
output.head(5)

To get back realistic-looking data, we can convert the concatenated column back into Name, City and District.

import pandas as pd

# Split the conatenated column by the separator and save the reuslts
names = []
countrycodes = []
districts = []

for x in output['concatenated']:
  try:
    name, countrycode, district = x.split('#')
  except:
    name, countrycode, district = [np.nan]*3
  names.append(name)
  countrycodes.append(countrycode)
  districts.append(district)

# Add the individual columns back in
output['Name'] = pd.Series(names)
output['CountryCode'] = pd.Series(countrycodes)
output['District'] = pd.Series(districts)

# Drop the concatenated column
output.drop(labels=['concatenated'], axis=1, inplace=True)

As a result, the output now looks like our original data.

output.head()

Most importantly, the Name, CountryCode and District columns now make sense!

Caveats of transforming the data

The transform strategy is an efficient and elegant approach to modeling. But there is a downside: The transform strategy might lose some mathematical properties.

To see why, consider the model's perspective:

• Cambridge#GBR#England is completely different from
• Cambridge#USA#Massachusetts is completely different from
• Boston#USA#Massachusetts

The problem is that two of these actually have something in common -- they are located in Massachusetts, USA. So the model will not be able to learn anything special about Massachusetts or USA as a whole.

As an example, let's see how well the model was able to learn populations of US-based cities.

import matplotlib.pyplot as plt

# Populations of real US cities
real_usa = data.loc[data['CountryCode'] == 'USA', 'Population']

# Populations of synthetic US cities
synth_usa = output.loc[output['CountryCode'] == 'USA', 'Population']

# Plot the distributions
plt.ylabel('US City Data')
plt.xlabel('Population')
_ = plt.boxplot([real_usa, synth_usa],
                showfliers=False,
                labels=['Real', 'Synthetic'],
                vert=False
)
plt.show()

The real data shows less variation in city population than the synthetic data. The differences make sense because our model wasn't able to learn about the USA as one complete concept.

Can we fix this? It's challenging to fix this issue without degrading the mathematical correlations in some other way. If you have any ideas, we welcome you to join our discussion!

Inputting a UniqueCombination into the SDV

We built the constraint -- both the reject_sampling and transform approaches -- directly into the SDV library. If you have sdv installed, this is ready to use. Import the UniqueCombinations class from the constraints module.

from sdv.constraints import UniqueCombinations

# Create a Unique Combinations constraint
unique_city_country_district = UniqueCombinations(
  columns=['Name', 'CountryCode', 'District'],
  handling_strategy='transform' # you can change this 'reject_sampling' too
)

# Create a new model using the constraint
updated_model = GaussianCopula(
  constraints=[unique_city_country_district],
  categorical_transformer='label_encoding'
)

Now, you can train the model on your data and sample synthetic data.

np.random.seed(35)

updated_model.fit(data)
updated_model.sample(5)

All of the synthetic data is guaranteed to follow the UniqueCombinations constraint.

Takeaways

1. We can identify a UniqueCombinations requirement by asking: Should it be possible to further mix-and-match the data?
2. We can enforce any logical constraint by using reject sampling, which throws out any invalid data. This is not efficient for UniqueCombinations.
3. An alternative approach is to transform the data, forcing the ML model to learn the constraint. For UniqueCombinations we transformed the data by concatenating it.
4. The logic for UniqueCombinations is already built into the SDV's constraints module, and is ready to use.

Further reading:

• Engineering Constraints Blog Article
• Handling Constraints User Guide
• Tabular Constraints API

We started SDV open source in 2018 at MIT with the goal of creating a powerful, usable, machine learning-based synthetic data generation software system. The core belief that drove us was the conviction that more than 90% of data work can be done using synthetic data instead of real data. Early experiments at MIT had been promising and we were ready to invest our time and energy into that promise.

Now, 3 years later, we are pleased to see that the market demand for synthetic data is increasing. In a 2021 article, Gartner predicted that 60% of data used for AI & analytics will be synthetic by 2024.

As time progressed, we used feedback from our users to make numerous improvements to the SDV (see articles Part 1 and Part 2). In response, we've seen increased usage, validating the market need for synthetic data generation software. In this article, we'll describe the SDV growth trends in detail.

Persistent 4x/year growth in downloads

Every year we are experiencing a 4x increase in SDV downloads. In 2021, we had 135,000 downloads of SDV – up from 30,576 in 2020. From the start of 2020 to the end of 2021, we have seen 16x total increase in SDV downloads. The figure below shows our yearly usage.

The downloads are coming from all over the world. In the map below, we list the top 10 countries.

Why are users downloading the SDV? We know that they want to create synthetic data, but they are using the synthetic data to solve a variety of different needs. We will explore this more and share it in a future article.

Over a thousand new community members

Another measure of our growth – and validation from the market – comes from the SDV community we've built on our GitHub and Slack. In 2021, we welcomed more than 1000 new members to these spaces.

As this article points out, members contribute in several different ways: Many help increase awareness of an open source solution for this enterprise pain point. Meanwhile, others jump in, use it and give feedback actively. In 2021, we doubled the number of unique users raising issues on our GitHub. Throughout the  year, over 200 members actively participated in our forums by raising GitHub issues or contributing to discussions on Slack.

Enterprise feedback is particularly useful to us. This type of feedback comes from users who are solving targeted business problems with the SDV. Direct and succinct feedback explains what would make the SDV more useful. An example is shown below.

Our team addresses the user feedback throughout the entire SDV ecosystem. The ecosystem includes not only modeling, but also the ability to compare models through SDGym and measure synthetic data quality through SDMetrics. In 2021, the team put out 49 releases throughout the SDV ecosystem, doubling our number of releases in 2020.

Looking forward to 2022!

We are looking forward to 2022! With so many users giving us feedback, we have a long list of features that we want to incorporate. We can't wait to share with our community what everyone is using SDV for, and keep on climbing to our original goal: 90% of data work accomplished with synthetic data.

The SDV uses machine learning (ML) to automatically learn rules (aka correlations) from real data and generate accurate synthetic data. While these models are powerful, they may not learn everything. In our previous article, we described how the SDV models may not learn deterministic rules. These are patterns and laws that are inherent to the dataset:

• They are unchangeable, no matter what data you input.
• They describe rules that must apply to every row, no exceptions.

Luckily, it's possible for you to improve the machine learning model: When you input constraints, it ensures the model will learn deterministic rules and ultimately improve the quality of your synthetic data.

In this article, we'll dive into the technical details of how you can apply constraints and how they work under-the-hood. You can also follow along in our notebook.

!pip install sdv==0.13.0

import numpy as np
import warnings

warnings.filterwarnings('ignore')

The Dataset

The dataset we're using comes from a Kaggle Competition hosted by Expedia. We've modified the data slightly for our use.

from sdv.demo import load_tabular_demo

data = load_tabular_demo('expedia_hotel_logs')

In this real-world dataset, each row represents a search result for a hotel booking.

For the purposes of this notebook, we'll drop some columns that aren't useful to us.

import pandas as pd

# Drop some columns that aren't useful for this demo
drop_columns = ['date_time', 'user_location_country', 'user_location_region',
                'user_location_city', 'user_id', 'srch_destination_id',
                'hotel_country', 'hotel_market', 'hotel_cluster',
                'srch_destination_type_id', 'orig_destination_distance',
                'posa_continent', 'site_name', 'channel']
data = data.drop(drop_columns, axis=1)

# make sure these columns are read as datetimes
for col in ['srch_ci', 'srch_co']:
  data[col] = pd.to_datetime(data[col])

# Inspect the data
data.head()

The search parameters, for finding a hotel room, saved in this dataset come from from user's input. For example:

Deterministic Rule

In order for the search to be valid, the searched check-in date must happen before the searched check-out date. That is: srch_ci < srch_co.

This is an inherent property of any search, not just for this particular dataset -- we call this a deterministic rule. We can verify if this is true by checking for any exceptions.

print('Violations of the deterministic rule')
len(data[data['srch_ci'] > data['srch_co']])

0

Will SDV's machine learning model learn this out of the box?

To test this, let's use SDV to learn a GaussianCopula model from the data and sample synthetic data.

from sdv.tabular import GaussianCopula

np.random.seed(0)

model = GaussianCopula(primary_key='log_id')
model.fit(data)

synth_data = model.sample(500)
synth_data.head()

Now, we can inspect the synthetic data to see if there are any invalid rows.

invalid_row_indices = synth_data['srch_ci'] > synth_data['srch_co']
invalid_rows = synth_data[invalid_row_indices]

num_invalid = len(invalid_rows)
perc_invalid = num_invalid / len(synth_data) * 100
print('Number of invalid rows:', num_invalid, '(', round(perc_invalid, 2), '%)')

invalid_rows.head()

The majority of the rows (94.8%) are valid, meaning the model learned the rule pretty accurately. It learned probabilistically that if the srch_ci is higher srch_co should be even higher. However, some invalid rows (~5%) are still created so the model did not learn this deterministic rule.

This raises the question: What can we do to enforce a deterministic rule?

Improving the synthetic data

Let's explore some options for enforcing our deterministic rule in order to improve the overall quality of the synthetic data.

Rejecting invalid data

The simplest solution is to simply drop the invalid rows, and continually sample from the model until the desired amount of valid rows are produced. We call this reject sampling.

The code below performs reject sampling until we have synthesized 500 rows.

import pandas as pd

# Keep track of how many valid rows we've sampled
num_valid_rows = synth_data.shape[0] - invalid_rows.shape[0]

while num_valid_rows < 500:
  # Reject the invalid data 
  synth_data = synth_data.drop(invalid_rows.index)
  
  # Create new data to replace the invalid data
  new_data = model.sample(500-num_valid_rows)
  synth_data = pd.concat([synth_data, new_data])
  invalid_rows = synth_data[synth_data['srch_ci'] > synth_data['srch_co']]
  num_valid_rows = synth_data.shape[0] - invalid_rows.shape[0]

synth_data.reset_index(drop=True, inplace=True)

Now, there are no invalid rows in our dataset.

invalid_rows = synth_data[synth_data['srch_ci'] > synth_data['srch_co']]
invalid_rows.shape[0]

0

In this example, we got lucky. Only a small percentage of the rows were invalid each time sample was called.

What would happen if majority of the rows were invalid every time we sampled? It would take a longer time to get all the desired rows. Sampling time is the primary drawback of reject sampling. Is there another approach we can use to improve the time?

Transforming your data

Instead of reject sampling, what if the model never produced invalid rows in the first place? To achieve this, we can alter the input data to the model so it's forced to learn the constraint.

Let's stop giving the srch_ci and srch_co to the model. Instead, let's teach the model to learn the srch_ci and the difference between the dates.

difference = srch_co - srch_ci

The model will produce srch_ci and difference as a result. Then, we can re-compute srch_co with the opposite formula.

srch_co = srch_ci + difference

(Of course, we need to make sure the difference is always positive, which we can do using a log + 1.)

Let's see this in action.

# Compute the difference
diff = (data['srch_co'] - data['srch_ci']).astype('timedelta64[D]')

# Take the log and add one to ensure that it's positive
date_diff = np.log(diff + 1)

# The model should learn this column instead of the checkout date
modified_data = data.drop('srch_co', axis=1)
modified_data['difference'] = date_diff
modified_data[['srch_ci', 'difference']].head()

Now, we can fit the model with the modified data. The new samples will include the srch_ci and date_diff columns.

np.random.seed(20)

modified_model = GaussianCopula(primary_key='log_id')
modified_model.fit(modified_data)

modified_synth_data = modified_model.sample(500)
modified_synth_data[['srch_ci', 'difference']].head()

We can recompute the srch_co based on srch_ci and difference.

# Undo the log+1 that we added
diff = (np.exp(modified_synth_data['difference'].values).round() - 1).clip(0).astype('timedelta64[ns]')

# Reconstruct the end_date and remove the date_diff column
modified_synth_data['srch_co'] = modified_synth_data['srch_ci'] + diff
modified_synth_data = modified_synth_data.drop('difference', axis=1)

modified_synth_data.head()

Let's verify that this computation does not create any invalid rows.

invalid_rows = modified_synth_data[modified_synth_data['srch_ci'] > modified_synth_data['srch_co']]
invalid_rows.shape[0]

0

The transformation worked! In our case, this was a more efficient way to enforce the deterministic rule.

But if our rule were more complex -- and we couldn't think of a transformation -- we could always fall back to reject sampling.

Inputting deterministic rules in the SDV

We've seen how reject sampling and transform can be used to improve the quality of the synthetic data by accounting for deterministic rules. However, it may be cumbersome for you to manually implement these strategies. In fact, we saw some common problems in our SDV user community:

• Users had multiple deterministic rules in their dataset. For example, there could be multiple comparisons between different pairs of columns.
• Users from multiple domains often had the same kind of deterministic rule. For example, one column being greater than another is a common deterministic rule, agonistic of a use case or domain.

To solve these problems, we introduced a constraints module in the SDV. With the constraints module, SDV users can easily input deterministic rules. Let's look at an example.

Using the SDV constraints module

The constraints module in the SDV contains several different types of pre-defined deterministic rules.

We will use the GreaterThan constraint, which will enforce that one column's values are always greater than another's.

from sdv.constraints import GreaterThan

Next, we can input the logic of our deterministic rule by creating a constraint object. The GreaterThan constraint accepts the column names as input.

gt_constraint = GreaterThan(
  low='srch_ci',
  high='srch_co')

Finally, we can input this constraint when instantiating the model.

np.random.seed(10)

# Apply the constraint to the model
model_with_constraint = GaussianCopula(
  primary_key='log_id',
  constraints=[gt_constraint])

model_with_constraint.fit(data)

# Sample synthetic data
constrained_data = model_with_constraint.sample(500)
constrained_data.head()

As a result, we should see that all 500 generated rows are valid on the first try. No invalid rows are present in our dataset.

invalid_rows = constrained_data[constrained_data['srch_ci'] > constrained_data['srch_co']]
invalid_rows.shape[0]

0

Using the SDV was much simpler than writing the code ourselves! Plus, we can create multiple constraints for the same dataset an easily use them on other datasets.

Specifying the strategy in the constraints module

By default, the GreaterThan constraint uses the transform strategy. However, you can use the handling_strategy argument to control this. This argument accepts 'reject_sampling' or 'transform' as valid strategies.

gt_reject_constraint = GreaterThan(
  low='srch_ci',
  high='srch_co',
  handling_strategy='reject_sampling' # specify the strategy
)

Similar to before, we can then input this constraint into the model.

np.random.seed(30)

# Apply the constraint to the model
model_with_reject_constraint = GaussianCopula(
  primary_key='log_id',
  constraints=[gt_reject_constraint])

model_with_reject_constraint.fit(data)

# Sample synthetic data
constrained_reject_data = model_with_reject_constraint.sample(500)
constrained_reject_data.head()

invalid_rows = constrained_reject_data[constrained_reject_data['srch_ci'] > constrained_reject_data['srch_co']
invalid_rows.shape[0]

0

What other deterministic rules are already available in SDV?

The GreaterThan constraint is one kind of deterministic rule, but there may be others that apply to your dataset. The SDV offers more constraints for other types of logic.

• Unique when values in a column must be unique to the entire dataset.
• UniqueCombinations to limit the permutations between multiple columns.
• Positive and Negative to enforce boundaries.
• ColumnFormula when there is a formulaic association between columns.
• Rounding to enforce decimal precision.
• Between when one column's values must be between 2 other values.
• OneHotEncoding when your data includes a variable with one hot encoding.

For each of them, you can specify handling strategies for reject_sampling to discard invalid data or transform to modify the data (unique to each constraint).

What if my rule isn't included in the module?

You may come across a rule that cannot be described by any of the constraints classes in the SDV. In this case, you can define a CustomConstraint with logic specific to your use case.

Additionally, consider filing a feature request on GitHub with details about your use case & scenario. We can add your logic as a pre-defined constraint so others can benefit from it too!

Takeaways

In this notebook, we explored what happens when we have a deterministic rule in our dataset.

1. Machine learning models may not able to learn the deterministic rules out of the box, but it is possible to improve the model to learn these types of rules.
2. Deterministic rules can be handled by discarding invalid data (reject sampling) or by adding some clever preprocessing to your code (transforming).
3. The SDV offers a constraints module that allows you to input commonly found deterministic rules. You can specify the handling strategy for each constraint and apply multiple rules to the same dataset.

Further Reading

For further information about constraints refer to the Handling Constraints User Guide.

In our previous article, we explored how machine learning (ML) plays a key role in synthetic data creation. One of the biggest strengths of ML is automatic rule detection (also known in ML terms as correlations): The algorithms are designed to learn patterns in the data, even without additional user input. The result is synthetic data that resembles the original, right down to its mathematical properties!

However, in some cases, applying an ML model right out of the box may not immediately achieve the desired result. In this article, we'll explore the strengths of ML models and go through those areas where user input may be required.

Strengths of ML Models

The goal of any ML-based synthetic data generation software is to learn from and emulate the input data. To illustrate this, let's pretend you work in the car insurance business, and you're in possession of a real dataset related to drivers and their insurance:

An ML-based system, such as the Synthetic Data Vault (SDV), will learn patterns from the real data and use it to create new synthetic data. Recall some of the important patterns that ML algorithms detect:

• Shapes. The general shape of the data. For example, in the dataset above, 50% of drivers have Collision Coverage and the Annual Premium is uniformly scattered between $3,000 and $9,000.
• Correlations. The trends within the data. For example, having Collision Coverage -- especially Standard coverage -- means a higher Annual Premium.

These shapes and correlations will be present in the synthetic data that is outputted by the ML model, as shown below.

Perhaps the single biggest strength of an ML algorithm is its ability to learn rules by looking for general patterns in the data, using probability and statistics.

What ML models do not learn out of the box

Let's take a closer look at the synthetic car insurance data. You might notice that two of the rows in the synthetic data don't make complete sense. Below, we've highlighted the errors.

Do you see what has gone wrong? In the first row, the license expired 3 years earlier than it was issued. In the last row, a driver without Collision Coverage has a Collision Policy Type. Additionally, the same Customer ID has been repeated in Row 3 and Row 4.

There are three rules that the ML algorithm did not follow:

1. License Expiration > License Issue Year
2. If Has Collision Coverage = NO, then Collision Policy Type must be empty
3. All Customer IDs must be unique

Why does the ML model easily pick up on some rules and not others? To answer this question, we can look closely at the rules themselves. All of the rules that the ML model successfully learned -- including the distribution shapes and the correlations -- were based on general trends. These probabilistic rules apply to a majority of the relationships within the dataset, but not all of them. Although they have to make sense in aggregate, a few rows may be exceptions.

By contrast, the rules that the ML model failed to learn were stricter. These deterministic rules describe intrinsic laws of nature, time or logic. Each and every row must adhere to them, and they won't change regardless of  how much (or how little) data has been given to the ML model.

To continue with the driving theme: A probabilistic rule is like a yield sign, signaling a general recommendation that works out differently for each individual situation -- some cars may need to stop, while others just slow down. Meanwhile, a deterministic rule is like a stop sign, demanding that every single car must come to a full stop.

By default, our ML model assumed that all rules were probabilistic. When this happens, synthetic data still generally follows the desired properties -- for example, License Expiration > License Issue Year -- for most of the rows, but not for every row.

Improving the ML Models using constraints

Just because the ML model didn't automatically follow a deterministic rule doesn't mean that it can't. It's possible to improve the model so that it understands this type of rule. As a user working with the SDV, you can input deterministic rules into your model using constraints.

An ML model built using constraints will accommodate both probabilistic and deterministic rules.

Do you need SDV constraints?

Deterministic rules are often easy to spot in your dataset: They are the rules that every single row must follow in order to be valid, regardless of how much data there is overall.  But even if you identify the right constraints, there are some cases where you might not actually want to supply them to the SDV.

Because the SDV learns probabilistic rules, most of the synthesized data is generally valid. Having a few errors sprinkled in might actually be beneficial if you want your synthetic data to cover some edge cases. For example, if you're using the synthetic data to test insurance claim software, leaving in some weird data points might help you ensure that the software can handle unexpected cases -- like the License Expiration accidentally being set too early.

The figure below shows a few questions you can ask to determine whether adding a constraint is the right approach.

The SDV Constraints offering

If you decide that adding deterministic rules is important for generating your synthetic data, the SDV has many different constraints to choose from! The table below describes the constraints you would need in order to define the deterministic rules that would best mold your Car Insurance dataset.

The SDV offers many more possible constraints, including:

• UniqueCombinations
• Positive and Negative
• Rounding
• Between
• OneHotEncoding

You can add multiple constraints to the same dataset in order to accommodate all the deterministic rules you need. For more details, read the Constraints User Guide.

Takeaways

In this article, we learned that:

• Data is governed by rules. The SDV automatically learns probabilistic rules, which describe overall trends or patterns in the data.
• However, sometimes the data has deterministic rules, which are always inherent no matter how much or how little data there is. ML-based systems, including the SDV, may not enforce deterministic rules out of the box.
• Users can input deterministic rules to the SDV using constraints. To figure out whether you should input a constraint, ask yourself whether there are any rules that the data must always follow. There are many constraints to choose from.

In future articles, we'll dive deeper into this topic. We'll explore the technical details behind constraints, and how exactly the SDV's ML models are able to learn deterministic rules.

Data is a great source of information. Real data — which is based on observations of real-world phenomena like weather, movements on a factory floor or the activities of a user base — can help us notice trends, increase business efficiency and solve problems.

But data can be helpful even if it isn’t real. This data, sometimes called fake or test data, doesn’t come directly from real-world observations, but is instead artificially crafted by a human or machine. The latest and most complex iteration of this data type — what we call synthetic data — builds on previous work done in this space.

In this article, we'll go through the history of fake data. By the end, you'll be able to answer the following questions:

• What were the original motivations and tools for manually creating data?
• What differentiates synthetic data from other types of fake data?
• What role does machine learning play in generating synthetic data?

The Dawn of Fake Data: Test Data Management

One group of people has worked with fake data for a long time: software engineers. They need data in order to test the systems they build, and the real stuff isn't always usable (for example, due to privacy).

Let's pretend it's the early 2000s, and you're an IT professional working at a bank. You're responsible for the software that updates account balances after each transaction. You'd like to test this software before putting it into production. What do you do?

Most likely, you'll come up with a few test scenarios to ensure that your functionality — updating the balance — can properly handle a variety of inputs.

Notice that in order to create these scenarios, you had to generate data: various starting balances ($500, $20, $10) as well as a transfer amount ($20). This is an early version of using fake data in order to test your software!

Using Tools for Manual Creation

Now let's fast forward in time. Over the years, your software has gotten even more complex, and you're constantly adding new functionalities. For example, maybe you start allowing transfers with foreign currency.

You need to test these functionalities before you roll them out. To save time, you might end up using -- or creating -- a tool that allows you to generate and manage fake data for testing.

The simplest tool may be a basic permutation, as illustrated below.

A more sophisticated tool might allow you greater control over the rules the data must follow. It will also allow you to create more columns as your functionalities increase. For example, maybe the bank now offers two different account types: Premium and Normal.

Now you need a test data generation tool that can handle all of these variables and come out with something like this:

Many test data management tools use sophisticated logic to precisely create these data columns and their values. But the rules they use are manually written, and rely on human intuition and domain knowledge. For example:

• Account type = Premium 10% of the time and Normal 90% of the time
• Starting balance is between $10,000 and $250,000 if Account type = Premium
  or between -$1,000 and $10,000 if Account type = Normal
  
• Transfer amount follows a bell curve with a mean of $7,500 and standard deviation of $1,000
• Etc.

There are downsides to this manual approach. It takes time and effort to come up with these rules, to keep track of them, and to update them as your application changes.

Adding Machine Learning

Adopting machine learning (ML) opens up entirely new avenues in data generation. In the process, it gets rid of some of these downsides.

At a high level, ML-based software (such as the Synthetic Data Vault) works in three steps:

1. The user inputs real data into the ML software
2. The ML software automatically learns patterns in the data
3. The software outputs data that contains those patterns

Let's go back to our banking example to see how this works. It's now 2021 and you're using the SDV to generate your test data. You input all the transactions your bank has handled in the last week.

After modeling, the SDV outputs entirely new data that looks and behaves like the original. An illustration of this is shown below.

Notice that the output data contains many of the same properties as the original. The model learned all of the following information:

• Ranges & Categories. Transfers range from $5K to $10K. Bank accounts can be either premium or normal. Etc.
• Shapes. 10% of accounts are premium. Transfers follow a bell curve distribution with a mean of $7,500 and a standard deviation of $1,000. Etc.
• Correlations. Premium bank accounts tend to have higher balances ($10K to $250K) than normal accounts (-$1K to $10K).

In other words: while the old test data management tools required you to manually come up with rules, ML-based tools learn these rules automatically. Moreover, they can learn new information. For example, the ML picked upon a couple of extra correlations:

• Premium accounts are more likely to transfer foreign currency.
• Normal accounts are more likely to be overdrawn (transfer more than their current balance).

Using an ML-based data generation tool will help you ensure that your software is robust against these typical cases. And while manual data generation tools generate fake data, ML-based approaches generate what we call synthetic data.

Benefits of Synthetic Data

There are some clear advantages to using synthetic data over fake data, especially in software testing. Below, we've detailed a few.

• Saves time with automation. Because ML automatically learns patterns from the real data, there is no need to spend a lot of time coming up with and inputting rules. ML learns rules that you may even miss.
• Is usable by non-experts. Realistic fake data can only be generated by domain experts, who know the precise rules governing the dataset. However, anyone can generate synthetic data. All they have to do is input the real data and the ML software takes care of the rest!
• Increases adaptability. Applications and data will inevitably change over time. It's easy to update synthetic data as this happens, simply by retraining the ML model with newer data.

Benefits of synthetic data expand beyond software testing. The SDV Community is using synthetic data for an ever-increasing variety of tasks, including machine learning development, de-biasing datasets and scenario planning.

Key Takeaways

In this article, we surveyed numerous ways of creating and using data  that is not real. In particular, we learned that:

• Creating fake data is not a novel concept. Older generations of tools will output fake data when given an explicit list of rules. This is especially useful for software testing.
• Adding ML to this process is a newer evolution. Users input real data into the ML model, and it's able to automatically infer the rules. Data generated using ML-based systems is known as synthetic data.
• Synthetic data's key advantages include its automation and adaptability. The uses of synthetic data expand beyond software testing.

In future articles, we'll put ML models to the test! We'll uncover their strengths and weaknesses, and guide you through getting the most from synthetic data using the Synthetic Data Vault.

The Synthetic Data Vault (SDV) is a software system that allows users all over the world to input a dataset and generate synthetic data. The SDV was born out of academic research at MIT — but in 2018, we open-sourced it, so that people all over the world could use it.

Since then, we've been listening carefully to our community's feedback, making sure that we address any gaps between theoretical academic research and practical use. This article is the second in a multi-part series detailing recent improvements to the SDV that make it work in the real world. Here we'll discuss how we've amped up the data synthesis workflow. (For our previous discussion about how we've improved core models, see Part 1.)

What are workflows?

We open sourced the SDV not just to let users generate synthetic data, but also to allow them use that data to solve real-world problems. Our community taught us that actually using the SDV involves a multi-step process — and that improving the system means paying attention to this entire workflow, not just the core machine learning.

According to our users, this workflow boils down to a few generalizable steps:

1. Identifying real datasets that need to be synthesized
2. Transforming the datasets into a machine-readable format
3. Running the machine learning model
4. Synthesizing data according to particular specifications
5. Reversing the transformations such that the synthesized data looks like the original
6. Evaluating the synthesized data that results

These steps are illustrated in the diagram below.

The key insight from our users was that the application of machine learning models is only one step of a much larger puzzle. When the open source community helped us understand this, we were able to improve on the SDV software by adding in transformations, synthesizing options, and evaluation tools -- all detailed below.

Transforming Data

One major lesson from our open source community was how messy real-world datasets are compared to those used in academia. Academic datasets often come pre-sanitized and ready for numerical use. In the real world, however, databases are growing and changing constantly, and are often significantly different from the optimal yet theoretical structures used by machine learning researchers.

Two thorny data types frequently encountered in the real world are datetimes and null values.

• Datetimes can follow many different formats, including YYYY-MM-DD or MM-DD-YY. However, machine learning models accept numerical values only. Usually these are Unix timestamps, defined as the number of seconds that have elapsed since January 1, 1970. By this logic, a date like 2021-01-01 will transform into the number 1609488000.
• Null values also present a problem for mathematical models when they appear in numerical data. While users can tell models to ignore these values, the presence of a null might actually indicate something important, like a user declining to answer a question. To account for this, the SDV creates a new, binary column to address whether the original value is null.

To solve this problem, we introduced a new library called Reversible Data Transforms (RDT). The RDT library contains necessary logic for transforming different types of real world data to its machine-ready counterpart — as well as the logic for its reversal, so that a synthetic data user won't know the difference. The RDT is a standalone library that can reach beyond the synthetic data space, helping data scientists and academics across fields to clean their data. Since November 2020, the RDT has been supported on all major platforms including MacOS, Windows, and Linux.

Synthesizing Data Conditionally

When we first imagined the SDV, we assumed users would simply want to use all the synthetic data generated by the model. However, we soon found that some users have more complex needs, and require more control over the data they synthesize — opening up new possibilities for synthetic data in the process.

For example, one of our users, an engineer, found a whole new use for SDV. The engineer was writing a machine learning classifier on a dataset when they noticed it was unbalanced. Applying any algorithms to this dataset would lead to biased models. The engineer realized that, if used strategically, SDV could actually debias the data — if it only generated data with rarer attributes, the synthetic data it created could be combined with the real data to form a fully balanced dataset.

In February of 2021, we added conditional sampling to the SDV to enable this use case. Now, users can specify attributes or values that must be present in the synthesized data. In addition to debiasing datasets, users can use this feature to test hypothetical scenarios.

Evaluating Synthesized Data

When the entire system is working smoothly and outputting synthetic data, users still need to know: Is the data good enough to use? This vital question inspired us to add evaluation capabilities to the SDV. In doing so, we faced two key challenges: Defining the metrics, and creating a useful process.

Metrics

No single metric perfectly captures the different dimensions of synthetic data users may want to evaluate. Some want to preserve a high degree of mathematical likeness, others want to emphasize a particular column for machine learning predictions, and still others are more focused on threat models that can compromise privacy.

To address this, we created a separate library, SDMetrics, to define evaluation metrics. The library now includes a suite of metrics that cover differentiation of synthetic and real data, statistical likeness, and privacy.

Application

Rather than apply metrics on an ad-hoc basis, some SDV power users were creating mini-workflows to rapidly test out different models, datasets and evaluation criteria in succession. Inspired by their innovation, we created SDGym, a system that allows users to input models, datasets and success metrics to build a comprehensive evaluation framework.

The SDV Software Today

The SDV software is continuously evolving based on community feedback. In this article, we discussed improvements to the workflow surrounding synthetic data generation, including data transformations, sampling methods and evaluation tools. Earlier, in Part 1 of this series, we discussed the core synthetic data models themselves. In future blog articles, we plan to dig deeper into each of these areas, and to uncover new ones with you.

Like the SDV, this blog is a collaborative effort. Use our Slack to let us know which topics you'd like to hear more about. And as always, use GitHub to file technical issues with the system. Working together, we can make SDV the most trusted, transparent and comprehensive platform for synthetic data generation!

For other inquiries, please contact info@sdv.dev.

In our last post, we introduced the Synthetic Data Vault (SDV) — a software system that allows users to input a dataset and generate synthetic data. The SDV was born out of academic research at MIT — but in 2018, we open-sourced it, so that people all over the world could use it.

Since then, we've been listening carefully to our community's feedback, making sure that we address any gaps between theoretical academic research and practical use. This multi-part series details recent improvements we've made so that SDV works in the real world. In this article, we focus on the machine learning-based modeling techniques that form the core of the system, while Part 2 will cover the surrounding workflow.

What's in a model?

At its core, the SDV is a set of machine learning models designed to understand and mimic real world data. Once the SDV creates a particular model, developers can generate synthetic data by sampling it. For synthetic data to be successful, this generative model must be correct — but through discussions with our open source community, we realized that there is no such thing as a single, winning approach that works every time. Each dataset and use case is different.

Our solution is to provide choices, giving users all the necessary tools to make useful synthetic data for each new case at hand. Let's dive into three popular uses of the SDV where such options are available: Tabular models, sequential data and business logic.

More Options for Tabular Models

The earliest version of SDV was based on a classic statistical method: Gaussian Copulas. This model is transparent by definition. It allows us to understand and exert control over formulas in the model, notably the distributions of each variable. This can be especially useful for business applications, where data often follows predictable distributions. For example, wind speed is known to follow a Weibull distribution, biological measures like height usually follow normal distributions and credit default rates often follow exponential distributions.

Meanwhile, advances in the AI space had also produced a robust, alternative model for those willing to sacrifice transparency: A deep learning technique called Generative Adversarial Networks (GANs). GANs model complex processes that don't follow known formulas. While these models’ inner workings aren’t easily explained by humans, they produce highly accurate results. We created a GAN, called CTGAN, specifically for synthetic data. This black box model is especially good at figuring out complex correlations between variables in large datasets.

For a long time, SDV allowed users a choice between our Gaussian Copulas based model, called GaussianCopula, and CTGAN to model tabular data. While this choice provided some flexibility, our users reported they had a hard time choosing between such extreme alternatives. We wondered if a middle ground was possible: Could we specify distributions while also using GANs to identify complex correlations?

We couldn't find any model that fit both of these requirements, so we made our own! A key insight was that we could use Gaussian Copulas to understand the data and transform it before applying it to a GAN. The result is CopulaGAN, a hybrid model we released in October 2020.

CopulaGAN combines the human accessibility of Gaussian Copulas with the robust accuracy of GANs. This innovation provides users with a new choice: a hybrid approach.

The Special Case of Sequential Data

Another tricky case pointed out by our users involved sequential data. While sequential data is stored in a table, it is unlike a regular table in that its rows are linked together, usually by a time component. This use case is extremely frequent, especially in finance — any table with credit card transactions, stock prices, or payments is almost certainly sequential.

At the time, solutions treated sequential data as a case of general tabular modeling. After all, sequential data is inside a table. However, these solutions failed to incorporate the key information that makes sequential data unique: The relationships that exist between rows.

While considering this pain point, we recognized sequential data as an entirely new case that required its own unique set of modeling techniques. In October 2020, we released our DeepEcho library, which focuses entirely on sequential data. We also introduced our PAR model: a GAN approach made specifically for sequential data.

Encoding Business Logic using Constraints

Even with a plethora of modeling choices, it's vital to capture nuances in business logic while modeling synthetic data. This is due to differences in how humans and machines understand datasets.

Often, humans can easily glean the meaning of a dataset using context clues. Consider a table showing the names and ages of students and their legal guardians. A human will intuitively realize that a student must be younger than their guardian.

But will a machine understand the same rule? Because all of the SDV's models use statistics, they analyze trends generally — meaning that in this case, they will include a small possibility that a student could be older than their guardian. After all, is it totally out of the question that an older individual could enroll and list their child as their guardian? Either way, only a human expert can truly figure out what makes sense for this dataset!

To solve this pain point, SDV introduced the concept of constraints in July 2020. Constraints give users the ability to encode their business knowledge and expertise into an SDV model. In our example, they could specify that a guardian's age must be greater than the student's. Currently, the GreaterThan and UniqueCombination constraints allow for easy handling of common scenarios. We also provide a blanket CustomConstraint class, which gives users flexibility to capture more nuanced knowledge.

More Community Feedback

We believe that the more humans and machines can work together, the more efficient our processes can become. In this article, we explained how user feedback about the SDV led to new core modeling techniques and innovations — enabling a system that now provides a choice of multiple models, handles sequential data, and understands constraints. In Part 2, we will discuss similar feedback-driven innovations in the rest of the workflow.

Using SDV — and giving us feedback — fuels this rapid evolution. To start a discussion, please message us on Slack or file an issue on GitHub. Working together, we can make SDV the most trusted, transparent and comprehensive platform for synthetic data generation!

For other inquiries, please contact info@sdv.dev.

Hello world! We, the creators of MIT's Synthetic Data Vault, warmly welcome you to our official blog. Soon we'll be using this space to deep-dive into topics related to our libraries, and to unpack ideas in the synthetic data space. We're looking forward to exploring this exciting area with you.

But first, we want to properly introduce our project: The Synthetic Data Vault (SDV), an open source software ecosystem for generating synthetic data. In this post, we’ll explain why synthetic data is important, and tell the story of how we created the vault. We’ll also lay out what’s in store — and how you can get involved. Let’s get started with a brief overview.

Synthetic Data What?

Synthetic data is a bold new frontier in machine learning. It allows developers to share and use data more effectively.

It may seem counterintuitive, but although billions of gigabytes of data are produced every day, there are still huge gaps in what developers are actually able to use. Accessibility concerns, regulatory issues and imbalanced datasets can all keep experts from using data. This impedes progress in finance, health care and other domains.

Good synthetic data can fill these gaps. The SDV uses machine learning to analyze data. Then, it creates fully synthetic datasets that mimic the original. Although the synthetic data is entirely machine generated, it maintains the original format and mathematical properties. This makes synthetic data versatile. It can completely replace the existing data in a workflow, or it can supplement the data to enhance its utility. Already, our users have successfully used the SDV to augment datasets, test applications, remove bias and more.

A History of the SDV

Our story starts in 2013. In MIT's Laboratory for Information and Decision Systems (LIDS), we were working on general data science projects. We had developed new techniques, and we were excited to test them on real datasets. However, as soon as we asked for the data, we hit roadblocks. The process for getting access to data turned out to be much more complex than we anticipated, with many regulations and security red tape.

We wondered: What if we didn't need the real data in the first place? If we had synthetic data with the same mathematical properties as the original, it would be much easier for everyone to share and use.

In 2016, we released a paper describing the very first iteration of the SDV. It introduced a novel technique for synthesizing multi-table data, and included trials where data scientists successfully used synthetic data instead of real data for machine learning tasks. Related research to come out of the lab included CTGAN, a novel approach to generating synthetic data using deep learning.

After these successes in the research community, we decided to move beyond purely academic solutions. Synthetic data has the potential to solve real-world problems faced by people on all sides of data science: internal developers writing software, external contractors working offshore, 3rd party partners offering services and even the end users who create the data. After some pilot testing on enterprise applications, we open sourced our work in 2018, publishing sdv on PyPi for general use. Open sourcing offered ample opportunities for collaboration and customization. It allowed users all over the world to test the SDV in enterprise settings, and helped the SDV ecosystem evolve into a one-stop shop for synthetic data needs!

We listened to feedback and, as of today, have made 93 releases (across all our libraries), addressing 504 issues. We have been thrilled to see a burgeoning community of invested users using the SDV to solve problems. We've seen over 200K user downloads from PyPi, 400 stars in the SDV GitHub repository and 200 developers in our Slack channel. Our community is global and includes people in diverse roles: academics, data scientists, operations managers, engineers and more. We are continually learning from our community, and we're excited to bring new innovations to you!

Just the Beginning

Synthetic data has the potential to revolutionize the entire field of data science, allowing us to solve problems that once seemed untouchable. We want the Synthetic Data Vault to be the most trusted, transparent and comprehensive platform for synthetic data generation, but we can't do it without our users. It's our ever-growing open source community that allows us to quickly repair bugs, triage feature requests and improve to serve a variety of real-world needs.  

That’s where you come in. If you’re already a member of this community, we can’t thank you enough. And if you’d like to get involved, see below for ways to get started. Either way, watch this space for more nuanced discussions about synthetic data. We're excited to share what we've learned from you, and show how we are collectively improving the ecosystem. It’s time to open the vault!

Want more ways to get involved?

• Follow us on Twitter @sdv_dev for release announcements, blog updates and more
• Join our Slack community to meet other users, discuss synthetic data solutions and suggest topics for the blog
• Visit & star our GitHub repositories
• If you've successfully used the SDV for your project, share your experience and tag us

For other inquiries, please contact us at info@sdv.dev.

*Total download statistics per continent come from the Linehaul project using BigQuery, and include mirrors. Are you aware of more accurate ways to count Python package downloads? Let us know!