This release primarily addresses bugs and feature requests related to using constraints for the single-table models.
Users can now enforce scalar comparison with the existing GreaterThan constraint and apply 5 new constraints: OneHotEncoding, Positive, Negative, Between and Rounding.
Additionally, the SDV will now auto-apply constraints for rounding numerical values, and for keeping the data within the observed bounds.
All related user guides are updated with the new functionality.
Add OneHotEncoding Constraint - Issue #303 by @fealho
GreaterThan Constraint should apply to scalars - Issue #410 by @amontanez24
Improve GreaterThan constraint - Issue #368 by @amontanez24
Add Non-negative and Positive constraints across multiple columns- Issue #409 by @amontanez24
Add Between values constraint - Issue #367 by @fealho
Ensure values fall within the specified range - Issue #423 by @amontanez24
Add Rounding constraint - Issue #482 by @katxiao
Add rounding and min/max arguments that are passed down to the NumericalTransformer - Issue #491 by @amontanez24
GreaterThan constraint between Date columns rasises TypeError - Issue #421 by @amontanez24
GreaterThan constraint’s transform strategy fails on columns that are not float - Issue #448 by @amontanez24
AttributeError on UniqueCombinations constraint with non-strings - Issue #196 by @katxiao
Use reject sampling to sample missing columns for constraints - Issue #435 by @amontanez24
Ensure privacy metrics are available in the API docs - Issue #458 by @fealho
Ensure forumla constraint is called ColumnFormula everywhere in the docs - Issue #449 by @fealho
This release changes the way we sample conditions to not only group by the conditions passed by the user, but also by the transformed conditions that result from them.
Conditionally sampling on variable in constraint should have variety for other variables - Issue #440 by @amontanez24
This release improves the constraint functionality by allowing constraints and conditions
at the same time. Additional changes were made to update tutorials.
Not able to use constraints and conditions in the same time - Issue #379
Update benchmarking user guide for reading private datasets - Issue #427
This release broadens the constraint functionality by allowing for the ColumnFormula
constraint to take lambda functions and returned functions as an input for its formula.
It also improves conditional sampling by ensuring that any id fields generated by the
model remain unique throughout the sampled data.
The CTGAN model was improved by adjusting a default parameter to be more mathematically
Additional changes were made to improve tutorials as well as fix fragile tests.
Tutorials test sometimes fails - Issue #355
Duplicate IDs when using reject-sampling - Issue #331
by @amontanez24 and @csala
discriminator_decay should be initialized at 1e-6 but it’s 0 - Issue #401 by @fealho and @YoucefZemmouri
Tutorial typo - Issue #380 by @fealho
Request for sdv.constraint.ColumnFormula for a wider range of function - Issue #373 by @amontanez24 and @JetfiRex
This release brings new privacy metrics to the evaluate framework which help to determine
if the real data could be obtained or deduced from the synthetic samples.
Additionally, now there is a normalized score for the metrics, which stays between 0 and 1.
There are improvements that reduce the usage of memory ram when sampling new data. Also there
is a new parameter to control the reject sampling crash, graceful_reject_sampling, which if
set to True and if it’s not possible to generate all the requested rows, it will just issue a
warning and return whatever it was able to generate.
The Metadata object can now be visualized using different combinations of names and details,
which can be set to True or False in order to display only the table names with details or
without. There is also an improvement on the validation, which now will display all the errors
found at the end of the validation instead of only the first one.
This version also exposes all the hyperparameters of the models CTGAN and TVAE to allow a more
advanced usage. There is also a fix for the TVAE model on small datasets and it’s performance
with NaN values has been improved. There is a fix for when using
UniqueCombinationConstraint with the transform strategy.
Memory Usage Gaussian Copula Trained Model consuming high memory when generating synthetic data - Issue #304 by @pvk-developer and @AnupamaGangadhar
Add option to visualize metadata with only table names - Issue #347 by @csala
Add sample parameter to control reject sampling crash - Issue #343 by @fealho
Verbose metadata validation - Issue #348 by @csala
Missing the introduction of custom specification for hyperparameters in the TVAE model - Issue #344 by @imkhoa99 and @pvk-developer
This version adds conditional sampling for tabular models by combining a reject-sampling
strategy with the native conditional sampling capabilities from the gaussian copulas.
It also introduces several upgrades on the HMA1 algorithm that improve data quality and
robustness in the multi-table scenarios by making changes in how the parameters of the child
tables are aggregated on the parent tables, including a complete rework of how the correlation
matrices are modeled and rebuild after sampling.
Fix probabilities contain NaN error - Issue #326 by @csala
Conditional Sampling for tabular models - Issue #316 by @fealho and @csala
HMA1: LinAlgError: SVD did not converge - Issue #240 by @csala
This release introduces a few changes in the HMA1 relational algorithm to decrease modeling
and sampling times, while also ensuring that correlations are properly kept across tables
and also adding support for some relational schemas that were not supported before.
A few changes in constraints and tabular models also ensure that situations that produced
errors before now work without errors.
Fix unique key generation - Issue #306 by @fealho
Ensure tables that contain nothing but ids can be modeled - Issue #302 by @csala
Metadata visualization improvements - Issue #301 by @csala
Multi-parent re-model and re-sample issue - Issue #298 by @csala
Support datetimes in GreaterThan constraint - Issue #266 by @rollervan
Support for multiple foreign keys in one table - Issue #185 by @csala
SDMetrics version is updated to include the new Time Series metrics, which have also
been added to the API Reference and User Guides documentation. Additionally,
a few code has been refactored to reduce external dependencies and a few minor bugs
related to single table constraints have been fixed
Add timeseries metrics and user guides - Issue #289 by @csala
Add functions to generate regex ids - Issue #288 by @csala
Saving a fitted tabular model with UniqueCombinations constraint raises PicklingError -
Issue #286 by @csala
Constraints: handling_strategy='reject_sampling' causes 'ZeroDivisionError: division by zero' -
Issue #285 by @csala
'ZeroDivisionError: division by zero'
This release updates to the latest CTGAN, RDT and SDMetrics libraries to introduce a
new TVAE model, multiple new metrics for single table and multi table, and fixes
issues in the re-creation of tabular models from a metadata dict.
Upgrade to SDMetrics v0.1.0 and add sdv.metrics module - Issue #281 by @csala
Upgrade to CTGAN 0.3.0 and add TVAE model - Issue #278 by @fealho
Add dtype_transformers to Table.from_dict - Issue #276 by @csala
Fix Metadata from_dict behavior - Issue #275 by @csala
This version updates the dependencies and makes a few internal changes in order
to ensure that SDV works properly on Windows Systems, making this the first
release to be officially supported on Windows.
Apart from this, some more internal changes have been made to solve a few minor
issues from the older versions while also improving the processing speed when
processing relational datasets with the default parameters.
The distribution argument of the GaussianCopula has been renamed to field_distributions.
The HMA1 and SDV classes now use the categorical_fuzzy transformer by default instead of
the one_hot_encoding one.
GaussianCopula: rename distribution argument to field_distributions - Issue #237 by @csala
GaussianCopula: Improve error message if an invalid distribution name is passed - Issue #220 by csala
Import urllib.request explicitly - Issue #227 by @csala
TypeError: cannot astype a datetimelike from [datetime64[ns]] to [int32] - Issue #218 by @csala
Change default categorical transformer to categorical_fuzzy in HMA1 - Issue #214 by @csala
Integer categoricals being sampled as strings instead of integer values - Issue #194 by @csala
In this version a new family of models for Synthetic Time Series Generation is introduced
under the sdv.timeseries sub-package. The new family of models now includes a new class
called PAR, which implements a Probabilistic AutoRegressive model.
This version also adds support for composite primary keys and regex based generation of id
fields in tabular models and drops Python 3.5 support.
Drop python 3.5 support - Issue #204 by @csala
Support composite primary keys in tabular models - Issue #207 by @csala
Add the option to generate string id fields based on regex on tabular models - Issue #208 by @csala
Synthetic Time Series - Issue #142 by @csala
This version adds a new tabular model based on combining the CTGAN model with the reversible
transformation applied in the GaussianCopula model that converts random variables with
arbitrary distributions to new random variables with standard normal distribution.
The reversible transformation is handled by the GaussianCopulaTransformer recently added to RDT.
Add CopulaGAN Model - Issue #202 by @csala
This release moves the models and algorithms related to generation of synthetic
relational data to a new sdv.relational subpackage (Issue #198)
As part of the change, also the old sdv.models have been removed and now
relational model is based on the recently introduced sdv.tabular models.
In this release the sdv.evaluation module has been reworked to include 4 different
metrics and in all cases return a normalized score between 0 and 1.
Included metrics are:
This release fixes a couple of minor issues and introduces an important rework of the
User Guides section of the documentation.
Error Message: “make sure the Graphviz executables are on your systems’ PATH” - Issue #182 by @csala
Anonymization mappings leak - Issue #187 by @csala
In this release SDV gets new documentation, new tutorials, improvements to the Tabular API
and broader python and dependency support.
Complete list of changes:
New Documentation site based on the pydata-sphinx-theme.
New User Guides and Notebook tutorials.
New Developer Guides section within the docs with details about the SDV architecture,
the ecosystem libraries and how to extend and contribute to the project.
Improved API for the Tabular models with focus on ease of use.
Support for Python 3.8 and the newest versions of pandas, scipy and scikit-learn.
New Slack Workspace for development discussions and community support.
This release introduces a new concept of Constraints, which allow the user to define
special relationships between columns that will not be handled via modeling.
This is done via a new sdv.constraints subpackage which defines some well-known pre-defined
constraints, as well as a generic framework that allows the user to customize the constraints
to their needs as much as necessary.
Support for Constraints - Issue #169 by @csala
This release introduces a new subpackage sdv.tabular with models designed specifically
for single table modeling, while still providing all the usual conveniences from SDV, such
Seamless multi-type support
Missing data handling
Currently implemented models are:
GaussianCopula: Multivariate distributions modeled using copula functions. This is stronger
version, with more marginal distributions and options, than the one used to model multi-table
CTGAN: GAN-based data synthesizer that can generate synthetic tabular data with high fidelity.
Support for Multiple Parents - Issue #162 by @csala
Sample by default the same number of rows as in the original table - Issue #163 by @csala
Add benchmark - Issue #165 by @csala
Use SDMetrics for evaluation - Issue #159 by @csala
Improve metadata visualization - Issue #151 by @csala @JDTheRipperPC
Add Metadata Validation - Issue #134 by @csala @JDTheRipperPC
Add Metadata Visualization - Issue #135 by @JDTheRipperPC
Add path to metadata JSON - Issue #143 by @JDTheRipperPC
Use new Copulas and RDT versions - Issue #147 by @csala @JDTheRipperPC
Create sdv.models subpackage - Issue #141 by @JDTheRipperPC
Adapt evaluation to the different data types - Issue #128 by @csala @JDTheRipperPC
Extend load_demo functionality to load other datasets - Issue #136 by @JDTheRipperPC
Methods to generate Metadata from DataFrames - Issue #126 by @csala @JDTheRipperPC
compatibility with rdt issue 72 - Issue #120 by @csala @JDTheRipperPC
Error docstring sampler.__fill_text_columns - Issue #144 by @JDTheRipperPC
Reach 90% coverage - Issue #112 by @JDTheRipperPC
Review unittests - Issue #111 by @JDTheRipperPC
Time required for sample_all function? - Issue #118 by @csala @JDTheRipperPC
Add option to model the amount of child rows - Issue 93 by @ManuelAlvarezC
Add Evaluation Metrics - Issue 52 by @ManuelAlvarezC
Ensure unicity on primary keys on different calls - Issue 63 by @ManuelAlvarezC
executing readme: ‘not supported between instances of ‘int’ and ‘NoneType’ - Issue 104 by @csala
Add warnings when trying to model an unsupported dataset structure. GH#73
Add option to anonymize data. GH#51
Add support for modeling data with different distributions, when using GaussianMultivariate model. GH#68
Add support for VineCopulas as a model. GH#71
Improve GaussianMultivariate parameter sampling, avoiding warnings and unvalid parameters. GH#58
Fix issue that caused that sampled categorical values sometimes got numerical values mixed. GH#81
Improve the validation of extensions. GH#69
Update examples. GH#61
Replaced Table class with a NamedTuple. GH#92
Fix inconsistent dependencies and add upper bound to dependencies. GH#96
Fix error when merging extension in Modeler.CPA when running examples. GH#86
First release on PyPI.