Danger

You are looking at the documentation for an older version of the SDV! We are no longer supporting or maintaining this version of the software

GaussianCopula Model¶

In this guide we will go through a series of steps that will let you discover functionalities of the GaussianCopula model, including how to:

• Create an instance of a GaussianCopula.

• Fit the instance to your data.

• Generate synthetic versions of your data.

• Use GaussianCopula to anonymize PII information.

• Specify the column distributions to improve the output quality.

What is GaussianCopula?¶

The sdv.tabular.GaussianCopula model is based on copula funtions.

In mathematical terms, a copula is a distribution over the unit cube $${\displaystyle [0,1]^{d}}$$ which is constructed from a multivariate normal distribution over $${\displaystyle \mathbb {R} ^{d}}$$ by using the probability integral transform. Intuitively, a copula is a mathematical function that allows us to describe the joint distribution of multiple random variables by analyzing the dependencies between their marginal distributions.

Let’s now discover how to learn a dataset and later on generate synthetic data with the same format and statistical properties by using the GaussianCopula model.

Quick Usage¶

We will start by loading one of our demo datasets, the student_placements, which contains information about MBA students that applied for placements during the year 2020.

In [1]: from sdv.demo import load_tabular_demo

Out[3]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0       17264      M        67.00      91.00  Commerce        58.00    Sci&Tech            False                 0                55.0   Mkt&HR     58.80  27000.0    True 2020-07-23 2020-10-12       3.0
1       17265      M        79.33      78.33   Science        77.48    Sci&Tech             True                 1                86.5  Mkt&Fin     66.28  20000.0    True 2020-01-11 2020-04-09       3.0
2       17266      M        65.00      68.00      Arts        64.00   Comm&Mgmt            False                 0                75.0  Mkt&Fin     57.80  25000.0    True 2020-01-26 2020-07-13       6.0
3       17267      M        56.00      52.00   Science        52.00    Sci&Tech            False                 0                66.0   Mkt&HR     59.43      NaN   False        NaT        NaT       NaN
4       17268      M        85.80      73.60  Commerce        73.30   Comm&Mgmt            False                 0                96.8  Mkt&Fin     55.50  42500.0    True 2020-07-04 2020-09-27       3.0


As you can see, this table contains information about students which includes, among other things:

• Their id and gender

• Their work experience

• The salary that they were offered

• The duration and dates of their placement

You will notice that there is data with the following characteristics:

• There are float, integer, boolean, categorical and datetime values.

• There are some variables that have missing data. In particular, all the data related to the placement details is missing in the rows where the student was not placed.

Let us use the GaussianCopula to learn this data and then sample synthetic data about new students to see how well the model captures the characteristics indicated above. In order to do this you will need to:

• Import the sdv.tabular.GaussianCopula class and create an instance of it.

• Call its fit method passing our table.

• Call its sample method indicating the number of synthetic rows that you want to generate.

In [4]: from sdv.tabular import GaussianCopula

In [5]: model = GaussianCopula()

In [6]: model.fit(data)


Note

Notice that the model fitting process took care of transforming the different fields using the appropriate Reversible Data Transforms to ensure that the data has a format that the GaussianMultivariate model can handle.

Generate synthetic data from the model¶

Once the modeling has finished you are ready to generate new synthetic data by calling the sample method from your model passing the number of rows that we want to generate. The number of rows (num_rows) is a required parameter.

In [7]: new_data = model.sample(num_rows=200)


This will return a table identical to the one which the model was fitted on, but filled with new data which resembles the original one.

In [8]: new_data.head()
Out[8]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0       17452      M        45.10      48.86  Commerce        52.12   Comm&Mgmt             True                 1               50.38   Mkt&HR     56.01      NaN   False        NaT        NaT       NaN
1       17358      M        62.50      62.68  Commerce        60.66   Comm&Mgmt            False                 0               81.70  Mkt&Fin     65.68      NaN   False        NaT        NaT       NaN
2       17457      F        88.33      85.19   Science        79.93      Others            False                 0               79.14   Mkt&HR     63.22  25356.0    True 2020-06-24 2020-12-21       5.0
3       17451      M        55.88      54.12   Science        52.92   Comm&Mgmt             True                 1               74.55   Mkt&HR     62.53      NaN   False        NaT        NaT       NaN
4       17393      F        83.34      66.42  Commerce        63.16   Comm&Mgmt            False                 0               78.91   Mkt&HR     53.27  20331.0    True 2020-03-07 2020-04-21       3.0


Note

There are a number of other parameters in this method that you can use to optimize the process of generating synthetic data. Use output_file_path to directly write results to a CSV file, batch_size to break up sampling into smaller pieces & track their progress and randomize_samples to determine whether to generate the same synthetic data every time. See the API section for more details.

In many scenarios it will be convenient to generate synthetic versions of your data directly in systems that do not have access to the original data source. For example, if you may want to generate testing data on the fly inside a testing environment that does not have access to your production database. In these scenarios, fitting the model with real data every time that you need to generate new data is feasible, so you will need to fit a model in your production environment, save the fitted model into a file, send this file to the testing environment and then load it there to be able to sample from it.

Let’s see how this process works.

Save and share the model¶

Once you have fitted the model, all you need to do is call its save method passing the name of the file in which you want to save the model. Note that the extension of the filename is not relevant, but we will be using the .pkl extension to highlight that the serialization protocol used is cloudpickle.

In [9]: model.save('my_model.pkl')


This will have created a file called my_model.pkl in the same directory in which you are running SDV.

Important

If you inspect the generated file you will notice that its size is much smaller than the size of the data that you used to generate it. This is because the serialized model contains no information about the original data, other than the parameters it needs to generate synthetic versions of it. This means that you can safely share this my_model.pkl file without the risk of disclosing any of your real data!

Load the model and generate new data¶

The file you just generated can be sent over to the system where the synthetic data will be generated. Once it is there, you can load it using the GaussianCopula.load method, and then you are ready to sample new data from the loaded instance:

In [10]: loaded = GaussianCopula.load('my_model.pkl')



Warning

Notice that the system where the model is loaded needs to also have sdv installed, otherwise it will not be able to load the model and use it.

Specifying the Primary Key of the table¶

One of the first things that you may have noticed when looking at the demo data is that there is a student_id column which acts as the primary key of the table, and which is supposed to have unique values. Indeed, if we look at the number of times that each value appears, we see that all of them appear at most once:

In [12]: data.student_id.value_counts().max()
Out[12]: 1


However, if we look at the synthetic data that we generated, we observe that there are some values that appear more than once:

In [13]: new_data[new_data.student_id == new_data.student_id.value_counts().index[0]]
Out[13]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
29        17417      M        74.29      64.34  Commerce        64.58   Comm&Mgmt             True                 1               94.90  Mkt&Fin     60.83  25123.0    True 2020-01-13 2020-05-23       6.0
34        17417      M        74.33      69.99  Commerce        69.98   Comm&Mgmt             True                 1               82.15  Mkt&Fin     69.62  21110.0    True 2020-04-30 2020-12-24       8.0
116       17417      M        55.66      65.41  Commerce        61.66   Comm&Mgmt            False                 0               63.27   Mkt&HR     56.65  20337.0    True 2020-01-15 2020-07-30       8.0
168       17417      M        66.39      67.81   Science        65.17   Comm&Mgmt             True                 1               63.60   Mkt&HR     56.04      NaN   False        NaT        NaT       NaN
179       17417      F        67.53      59.91  Commerce        66.61   Comm&Mgmt             True                 1               78.26  Mkt&Fin     57.99      NaN   False        NaT        NaT       NaN


This happens because the model was not notified at any point about the fact that the student_id had to be unique, so when it generates new data it will provoke collisions sooner or later. In order to solve this, we can pass the argument primary_key to our model when we create it, indicating the name of the column that is the index of the table.

In [14]: model = GaussianCopula(
....:     primary_key='student_id'
....: )
....:

In [15]: model.fit(data)

In [16]: new_data = model.sample(200)

Out[17]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0           0      M        70.15      56.61   Science        67.02    Sci&Tech             True                 1               84.79  Mkt&Fin     63.72  30139.0    True 2020-03-17 2020-05-24       4.0
1           1      M        62.97      73.01  Commerce        64.64   Comm&Mgmt            False                 0               87.81  Mkt&Fin     51.63  33673.0    True 2020-04-24 2020-09-03       4.0
2           2      F        60.11      72.13  Commerce        62.40    Sci&Tech            False                 0               54.86  Mkt&Fin     61.75      NaN   False        NaT        NaT       NaN
3           3      M        60.45      66.02   Science        66.42   Comm&Mgmt            False                 1               95.79  Mkt&Fin     58.74      NaN   False        NaT        NaT       NaN
4           4      M        79.08      64.88  Commerce        72.35   Comm&Mgmt            False                 0               54.22   Mkt&HR     54.68  21624.0    True 2020-02-21 2020-09-05       7.0


As a result, the model will learn that this column must be unique and generate a unique sequence of values for the column:

In [18]: new_data.student_id.value_counts().max()
Out[18]: 1


Anonymizing Personally Identifiable Information (PII)¶

There will be many cases where the data will contain Personally Identifiable Information which we cannot disclose. In these cases, we will want our Tabular Models to replace the information within these fields with fake, simulated data that looks similar to the real one but does not contain any of the original values.

Let’s load a new dataset that contains a PII field, the student_placements_pii demo, and try to generate synthetic versions of it that do not contain any of the PII fields.

Note

The student_placements_pii dataset is a modified version of the student_placements dataset with one new field, address, which contains PII information about the students. Notice that this additional address field has been simulated and does not correspond to data from the real users.

In [19]: data_pii = load_tabular_demo('student_placements_pii')

Out[20]:
student_id                                            address gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0       17264        70304 Baker Turnpike\nEricborough, MS 15086      M        67.00      91.00  Commerce        58.00    Sci&Tech            False                 0                55.0   Mkt&HR     58.80  27000.0    True 2020-07-23 2020-10-12       3.0
1       17265    805 Herrera Avenue Apt. 134\nMaryview, NJ 36510      M        79.33      78.33   Science        77.48    Sci&Tech             True                 1                86.5  Mkt&Fin     66.28  20000.0    True 2020-01-11 2020-04-09       3.0
2       17266        3702 Bradley Island\nNorth Victor, FL 12268      M        65.00      68.00      Arts        64.00   Comm&Mgmt            False                 0                75.0  Mkt&Fin     57.80  25000.0    True 2020-01-26 2020-07-13       6.0
3       17267                   Unit 0879 Box 3878\nDPO AP 42663      M        56.00      52.00   Science        52.00    Sci&Tech            False                 0                66.0   Mkt&HR     59.43      NaN   False        NaT        NaT       NaN
4       17268  96493 Kelly Canyon Apt. 145\nEast Steven, NC 3...      M        85.80      73.60  Commerce        73.30   Comm&Mgmt            False                 0                96.8  Mkt&Fin     55.50  42500.0    True 2020-07-04 2020-09-27       3.0


If we use our tabular model on this new data we will see how the synthetic data that it generates discloses the addresses from the real students:

In [21]: model = GaussianCopula(
....:     primary_key='student_id',
....: )
....:

In [22]: model.fit(data_pii)

In [23]: new_data_pii = model.sample(200)

Out[24]:
student_id                                            address gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0           0  26682 Gabrielle Villages\nPhilipchester, OH 87032      M        69.79      73.01  Commerce        68.67   Comm&Mgmt             True                 1               86.83  Mkt&Fin     58.08  22305.0    True 2020-04-09 2020-06-29       4.0
1           1           725 Amber Path\nNew Tonyaville, NJ 22954      M        59.91      55.95  Commerce        60.15    Sci&Tech            False                 0               73.94   Mkt&HR     59.15      NaN   False        NaT        NaT       NaN
2           2  67904 Bowen Ways Suite 290\nHernandezville, VT...      F        80.64      76.53  Commerce        67.01   Comm&Mgmt            False                 0               61.47  Mkt&Fin     75.94  23572.0    True 2020-02-09 2020-08-09       5.0
3           3     2313 Bryce Court Apt. 997\nJonesfurt, OK 62601      M        55.02      50.44   Science        58.49   Comm&Mgmt            False                 0               88.22  Mkt&Fin     55.82      NaN   False        NaT        NaT       NaN
4           4      793 Christopher Branch\nRonaldville, MD 38677      M        65.01      69.89   Science        59.87    Sci&Tech            False                 1               92.26  Mkt&Fin     62.11  36279.0    True 2020-03-07 2020-09-10       6.0


More specifically, we can see how all the addresses that have been generated actually come from the original dataset:

In [25]: new_data_pii.address.isin(data_pii.address).sum()
Out[25]: 200


In order to solve this, we can pass an additional argument anonymize_fields to our model when we create the instance. This anonymize_fields argument will need to be a dictionary that contains:

• The name of the field that we want to anonymize.

• The category of the field that we want to use when we generate fake values for it.

The list complete list of possible categories can be seen in the Faker Providers page, and it contains a huge list of concepts such as:

• name

• country

• city

• ssn

• credit_card_number

• credit_card_expire

• credit_card_security_code

• email

• telephone

In this case, since the field is an address, we will pass a dictionary indicating the category address

In [26]: model = GaussianCopula(
....:     primary_key='student_id',
....:     anonymize_fields={
....:     }
....: )
....:

In [27]: model.fit(data_pii)


As a result, we can see how the real address values have been replaced by other fake addresses:

In [28]: new_data_pii = model.sample(200)

Out[29]:
student_id                                            address gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0           0  164 Heather Oval Suite 640\nWilliamport, AR 84719      M        76.47      73.59  Commerce        71.57   Comm&Mgmt             True                 1               80.53  Mkt&Fin     53.94  25658.0    True 2020-02-23 2020-08-15       7.0
1           1  27241 James Field Suite 334\nSmithchester, UT ...      F        77.44      79.80  Commerce        70.15   Comm&Mgmt            False                 0               95.38  Mkt&Fin     71.97  23032.0    True 2020-03-12 2020-04-27       3.0
2           2      8835 Mccarthy Mills\nSouth Jennifer, ND 29127      M        65.68      72.36   Science        65.44    Sci&Tech             True                 0               97.00  Mkt&Fin     66.57  38959.0    True 2020-04-22 2020-07-02       4.0
3           3  00295 Graham Summit Apt. 744\nBrandyville, HI ...      M        81.68      67.19   Science        66.98   Comm&Mgmt            False                 0               50.60   Mkt&HR     64.21  23823.0    True 2020-04-17 2020-10-15       6.0
4           4             679 Crystal Dale\nFisherberg, DE 80466      F        77.43      65.59  Commerce        80.05    Sci&Tech            False                 0               86.86  Mkt&Fin     68.09  25328.0    True 2020-01-20 2020-08-20       8.0


Which means that none of the original addresses can be found in the sampled data:

In [30]: data_pii.address.isin(new_data_pii.address).sum()
Out[30]: 0


Now that we have discovered the basics, let’s go over a few more advanced usage examples and see the different arguments that we can pass to our GaussianCopula Model in order to customize it to our needs.

Exploring the Probability Distributions¶

During the previous steps, every time we fitted the GaussianCopula it performed the following operations:

1. Learn the format and data types of the passed data

2. Transform the non-numerical and null data using Reversible Data Transforms to obtain a fully numerical representation of the data from which we can learn the probability distributions.

3. Learn the probability distribution of each column from the table

4. Transform the values of each numerical column by converting them to their marginal distribution CDF values and then applying an inverse CDF transformation of a standard normal on them.

5. Learn the correlations of the newly generated random variables.

After this, when we used the model to generate new data for our table using the sample method, it did:

1. Sample from a Multivariate Standard Normal distribution with the learned correlations.

2. Revert the sampled values by computing their standard normal CDF and then applying the inverse CDF of their marginal distributions.

3. Revert the RDT transformations to go back to the original data format.

As you can see, during these steps the Marginal Probability Distributions have a very important role, since the GaussianCopula had to learn and reproduce the individual distributions of each column in our table. We can explore the distributions which the GaussianCopula used to model each column using its get_distributions method:

In [31]: model = GaussianCopula(
....:     primary_key='student_id',
....:     enforce_min_max_values=False
....: )
....:

In [32]: model.fit(data)

In [33]: distributions = model.get_distributions()


This will return us a dict which contains the name of the distribution class used for each column:

In [34]: distributions
Out[34]:
{'gender': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'second_perc': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'high_perc': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'high_spec': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'degree_perc': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'degree_type': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'work_experience': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'experience_years': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'employability_perc': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'mba_spec': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'mba_perc': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'salary': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'salary.is_null': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'placed': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'start_date': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'start_date.is_null': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'end_date': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'end_date.is_null': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'duration': 'copulas.univariate.truncated_gaussian.TruncatedGaussian',
'duration.is_null': 'copulas.univariate.truncated_gaussian.TruncatedGaussian'}


Note

In this list we will see multiple distributions for each one of the columns that we have in our data. This is because the RDT transformations used to encode the data numerically often use more than one column to represent each one of the input variables.

Let’s explore the individual distribution of one of the columns in our data to better understand how the GaussianCopula processed them and see if we can improve the results by manually specifying a different distribution. For example, let’s explore the experience_years column by looking at the frequency of its values within the original data:

In [35]: data.experience_years.value_counts()
Out[35]:
0    141
1     65
2      8
3      1
Name: experience_years, dtype: int64

In [36]: data.experience_years.hist();


By observing the data we can see that the behavior of the values in this column is very similar to a Gamma or even some types of Beta distribution, where the majority of the values are 0 and the frequency decreases as the values increase.

Was the GaussianCopula able to capture this distribution on its own?

In [37]: distributions['experience_years']
Out[37]: 'copulas.univariate.truncated_gaussian.TruncatedGaussian'


It seems that it was not, as it rather thought that the behavior was closer to a Gaussian distribution. And, as a result, we can see how the generated values now contain negative values which are invalid for this column:

In [38]: new_data.experience_years.value_counts()
Out[38]:
0    109
1     88
2      3
Name: experience_years, dtype: int64

In [39]: new_data.experience_years.hist();


Let’s see how we can improve this situation by passing the GaussianCopula the exact distribution that we want it to use for this column.

Setting distributions for individual variables¶

The GaussianCopula class offers the possibility to indicate which distribution to use for each one of the columns in the table, in order to solve situations like the one that we just described. In order to do this, we need to pass a field_distributions argument with dict that indicates the distribution that we want to use for each column.

Possible values for the distribution argument are:

• gaussian: Use a Gaussian distribution.

• gamma: Use a Gamma distribution.

• beta: Use a Beta distribution.

• student_t: Use a Student T distribution.

• gaussian_kde: Use a GaussianKDE distribution. This model is non-parametric, so using this will make get_parameters unusable.

• truncated_gaussian: Use a Truncated Gaussian distribution.

Let’s see what happens if we make the GaussianCopula use the gamma distribution for our column.

In [40]: from sdv.tabular import GaussianCopula

In [41]: model = GaussianCopula(
....:     primary_key='student_id',
....:     field_distributions={
....:         'experience_years': 'gamma'
....:     },
....:     enforce_min_max_values=False
....: )
....:

In [42]: model.fit(data)


After this, we can see how the GaussianCopula used the indicated distribution for the experience_years column

In [43]: model.get_distributions()['experience_years']
Out[43]: 'copulas.univariate.gamma.GammaUnivariate'


And, as a result, we can see how the generated data now have a behavior which is closer to the original data and always stays within the valid values range.

In [44]: new_data = model.sample(len(data))

In [45]: new_data.experience_years.value_counts()
Out[45]:
0    195
1     17
2      2
3      1
Name: experience_years, dtype: int64

In [46]: new_data.experience_years.hist();


Note

Even though there are situations like the one shown above where manually choosing a distribution seems to give better results, in most cases the GaussianCopula will be able to find the optimal distribution on its own, making this manual search of the marginal distributions necessary on very little occasions.

Conditional Sampling¶

As the name implies, conditional sampling allows us to sample from a conditional distribution using the GaussianCopula model, which means we can generate only values that satisfy certain conditions. These conditional values can be passed to the sample_conditions method as a list of sdv.sampling.Condition objects or to the sample_remaining_columns method as a dataframe.

When specifying a sdv.sampling.Condition object, we can pass in the desired conditions as a dictionary, as well as specify the number of desired rows for that condition.

In [47]: from sdv.sampling import Condition

In [48]: condition = Condition({
....:     'gender': 'M'
....: }, num_rows=5)
....:

In [49]: model.sample_conditions(conditions=[condition])
Out[49]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0           0      M        78.55      55.09   Science        67.28   Comm&Mgmt             True                 0               79.22  Mkt&Fin     51.32  25219.0    True 2020-04-24 2020-07-24       4.0
1           1      M        66.84      51.02  Commerce        68.47   Comm&Mgmt             True                 1               84.59   Mkt&HR     60.98  21149.0    True 2020-02-04 2020-09-04       7.0
2           2      M        83.70      72.86  Commerce        71.83    Sci&Tech             True                 0               91.34  Mkt&Fin     65.89  49280.0    True 2020-03-19 2020-08-25       6.0
3           3      M        51.47      74.47  Commerce        62.81   Comm&Mgmt            False                 0               90.36  Mkt&Fin     64.19      NaN   False        NaT        NaT       NaN
4           4      M        72.46      57.91   Science        62.23      Others            False                 0               62.29   Mkt&HR     61.21      NaN   False        NaT        NaT       NaN


It’s also possible to condition on multiple columns, such as gender = M, 'experience_years': 0.

In [50]: condition = Condition({
....:     'gender': 'M',
....:     'experience_years': 0
....: }, num_rows=5)
....:

In [51]: model.sample_conditions(conditions=[condition])
Out[51]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc  salary  placed start_date end_date  duration
0           0      M        55.56      45.49   Science        58.75   Comm&Mgmt            False                 0               71.96   Mkt&HR     51.37     NaN   False        NaT      NaT       NaN
1           1      M        47.64      61.13   Science        56.31   Comm&Mgmt            False                 0               69.97  Mkt&Fin     51.39     NaN   False        NaT      NaT       NaN
2           2      M        53.27      42.16  Commerce        60.32   Comm&Mgmt            False                 0               66.61   Mkt&HR     51.68     NaN   False        NaT      NaT       NaN
3           3      M        58.59      61.40   Science        51.40    Sci&Tech            False                 0               64.91   Mkt&HR     56.53     NaN   False        NaT      NaT       NaN
4           4      M        60.31      51.40   Science        62.76   Comm&Mgmt            False                 0               92.09  Mkt&Fin     55.17     NaN   False        NaT      NaT       NaN


In the sample_remaining_columns method, conditions is passed as a dataframe. In that case, the model will generate one sample for each row of the dataframe, sorted in the same order. Since the model already knows how many samples to generate, passing it as a parameter is unnecessary. For example, if we want to generate three samples where gender = M and three samples with gender = F, we can do the following:

In [52]: import pandas as pd

In [53]: conditions = pd.DataFrame({
....:     'gender': ['M', 'M', 'M', 'F', 'F', 'F'],
....: })
....:

In [54]: model.sample_remaining_columns(conditions)
Out[54]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0           0      M        52.58      65.72   Science        77.92   Comm&Mgmt             True                 1               88.31  Mkt&Fin     58.56  34328.0    True 2020-01-06 2020-06-10       7.0
1           1      M        80.66      65.09   Science        61.31      Others             True                 0               81.98  Mkt&Fin     64.07  44274.0    True 2020-02-04 2020-08-11       6.0
2           2      M        51.37      62.56   Science        58.52      Others            False                 0               89.42   Mkt&HR     61.95      NaN   False        NaT        NaT       NaN
3           3      F        56.96      59.14   Science        60.02   Comm&Mgmt             True                 0               92.01  Mkt&Fin     56.26      NaN   False        NaT        NaT       NaN
4           4      F        86.00      77.50  Commerce        64.01   Comm&Mgmt            False                 0               58.19  Mkt&Fin     67.82  22919.0    True 2020-05-23 2020-08-01       3.0
5           5      F        63.96      70.21  Commerce        72.77   Comm&Mgmt            False                 0               90.10  Mkt&Fin     58.39  25632.0    True 2020-03-03 2020-08-03       6.0


GaussianCopula also supports conditioning on continuous values, as long as the values are within the range of seen numbers. For example, if all the values of the dataset are within 0 and 1, GaussianCopula will not be able to set this value to 1000.

In [55]: condition = Condition({
....:     'degree_perc': 70.0
....: }, num_rows=5)
....:

In [56]: model.sample_conditions(conditions=[condition])
Out[56]:
student_id gender  second_perc  high_perc high_spec  degree_perc degree_type  work_experience  experience_years  employability_perc mba_spec  mba_perc   salary  placed start_date   end_date  duration
0           0      M        69.32      69.58  Commerce         70.0    Sci&Tech            False                 0               80.52   Mkt&HR     72.64      NaN   False        NaT        NaT       NaN
1           1      F        59.01      53.38   Science         70.0   Comm&Mgmt            False                 0               81.50  Mkt&Fin     57.05  20340.0    True 2020-05-22 2020-08-30       5.0
2           2      M        80.14      69.85  Commerce         70.0    Sci&Tech             True                 0               67.86   Mkt&HR     60.01  33384.0    True 2020-04-13 2020-08-19       5.0
3           3      M        46.17      66.03   Science         70.0   Comm&Mgmt            False                 0               61.59  Mkt&Fin     69.55      NaN   False        NaT        NaT       NaN
4           4      M        68.94      69.23  Commerce         70.0   Comm&Mgmt            False                 0               59.73  Mkt&Fin     59.04      NaN   False        NaT        NaT       NaN


How do I specify constraints?¶

If you look closely at the data you may notice that some properties were not completely captured by the model. For example, you may have seen that sometimes the model produces an experience_years number greater than 0 while also indicating that work_experience is False. These types of properties are what we call Constraints and can also be handled using SDV. For further details about them please visit the Constraints guide.

Can I evaluate the Synthetic Data?¶

After creating synthetic data, you may be wondering how you can evaluate it against the original data. You can use the SDMetrics library to get more insights, generate reports and visualize the data. This library is automatically installed with SDV.

To get started, visit: https://docs.sdv.dev/sdmetrics/