Great Expectations renders Expectations to clean, human-readable documentation called Data Docs . These HTML docs contain both your Expectation Suites as well as your data validation results each time validation is run – think of it as a continuously updated data quality report.
We recommend you create a virtual environment and install ydata-synthetic and great-expectations by running the following command on your terminal.
install ydata-synthetic great-expectations
diff --git a/1.4/objects.inv b/1.4/objects.inv
deleted file mode 100644
index cae3614e..00000000
Binary files a/1.4/objects.inv and /dev/null differ
diff --git a/1.4/reference/api/preprocessing/base/index.html b/1.4/reference/api/preprocessing/base/index.html
deleted file mode 100644
index dd4d7bef..00000000
--- a/1.4/reference/api/preprocessing/base/index.html
+++ /dev/null
@@ -1,2148 +0,0 @@
-
-
-
-
-
- BaseProcessor - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- BaseProcessor
-
-
-
-
-
-
-
-
- Bases: ABC BaseEstimator TransformerMixin
-
-
-
This data processor works like a scikit learn transformer in with the methods fit, transform and inverse transform.
-Args:
- num_cols (list of strings):
- List of names of numerical columns.
- cat_cols (list of strings):
- List of names of categorical columns.
-
-
- Source code in ydata_synthetic/preprocessing/base_processor.py
- @typechecked
-class BaseProcessor ( ABC , BaseEstimator , TransformerMixin ):
- """
- This data processor works like a scikit learn transformer in with the methods fit, transform and inverse transform.
- Args:
- num_cols (list of strings):
- List of names of numerical columns.
- cat_cols (list of strings):
- List of names of categorical columns.
- """
-def __init__ ( self , num_cols : Optional [ List [ str ]] = None , cat_cols : Optional [ List [ str ]] = None ):
-self . num_cols = [] if num_cols is None else num_cols
-self . cat_cols = [] if cat_cols is None else cat_cols
-self . _num_pipeline = None # To be overriden by child processors
-self . _cat_pipeline = None # To be overriden by child processors
-self . _col_transform_info = None # Metadata object mapping inputs/outputs of each pipeline
-@property
-def num_pipeline ( self ) -> BaseEstimator :
- """Returns the pipeline applied to numerical columns."""
-return self . _num_pipeline
-@property
-def cat_pipeline ( self ) -> BaseEstimator :
- """Returns the pipeline applied to categorical columns."""
-return self . _cat_pipeline
-@property
-def types ( self ) -> Series :
- """Returns a Series with the dtypes of each column in the fitted DataFrame."""
-return self . _types
-@property
-def col_transform_info ( self ) -> SimpleNamespace :
- """Returns a ProcessorInfo object specifying input/output feature mappings of this processor's pipelines."""
-self . _check_is_fitted ()
-if self . _col_transform_info is None :
-self . _col_transform_info = self . __create_metadata_synth ()
-return self . _col_transform_info
-def __create_metadata_synth ( self ) -> SimpleNamespace :
-def new_pipeline_info ( feat_in , feat_out ):
-return SimpleNamespace ( feat_names_in = feat_in , feat_names_out = feat_out )
-if self . num_cols :
-num_info = new_pipeline_info ( self . num_pipeline . feature_names_in_ , self . num_pipeline . get_feature_names_out ())
-else :
-num_info = new_pipeline_info ([], [])
-if self . cat_cols :
-cat_info = new_pipeline_info ( self . cat_pipeline . feature_names_in_ , self . cat_pipeline . get_feature_names_out ())
-else :
-cat_info = new_pipeline_info ([], [])
-return SimpleNamespace ( numerical = num_info , categorical = cat_info )
-def _check_is_fitted ( self ):
- """Checks if the processor is fitted by testing the numerical pipeline.
- Raises NotFittedError if not."""
-if self . _num_pipeline is None :
-raise NotFittedError ( "This data processor has not yet been fitted." )
-def _validate_cols ( self , x_cols ):
- """Ensures validity of the passed numerical and categorical columns.
- The following is verified:
- 1) Num cols and cat cols are disjoint sets;
- 2) The union of these sets should equal x_cols;.
- Assertion errors are raised in case any of the tests fails."""
-missing = set ( x_cols ) . difference ( set ( self . num_cols ) . union ( set ( self . cat_cols )))
-intersection = set ( self . num_cols ) . intersection ( set ( self . cat_cols ))
-assert intersection == set (), f "num_cols and cat_cols share columns { intersection } but should be disjoint."
-assert missing == set (), f "The columns { missing } of the provided dataset were not attributed to a pipeline."
-# pylint: disable=C0103
-@abstractmethod
-def fit ( self , X : DataFrame ) -> BaseProcessor :
- """Fits the DataProcessor to a passed DataFrame.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
- Returns:
- self (DataProcessor): The fitted data processor.
- """
-raise NotImplementedError
-# pylint: disable=C0103
-@abstractmethod
-def transform ( self , X : DataFrame ) -> ndarray :
- """Transforms the passed DataFrame with the fit DataProcessor.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
- Returns:
- transformed (ndarray): Processed version of the passed DataFrame.
- """
-raise NotImplementedError
-# pylint: disable=C0103
-@abstractmethod
-def inverse_transform ( self , X : ndarray ) -> DataFrame :
- """Inverts the data transformation pipelines on a passed DataFrame.
- Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this DataProcessor.
- Can be used to revert transformations of training data or for synthetic samples.
- Returns:
- result (DataFrame):
- DataFrame with all performed transformations inverted.
- """
-raise NotImplementedError
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- cat_pipeline : BaseEstimator
- property
-
-
-
-
-
-
-
Returns the pipeline applied to categorical columns.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Returns a ProcessorInfo object specifying input/output feature mappings of this processor's pipelines.
-
-
-
-
-
-
-
-
-
- num_pipeline : BaseEstimator
- property
-
-
-
-
-
-
-
Returns the pipeline applied to numerical columns.
-
-
-
-
-
-
-
-
-
- types : Series
- property
-
-
-
-
-
-
-
Returns a Series with the dtypes of each column in the fitted DataFrame.
-
-
-
-
-
-
-
-
-
-
-
-
- fit ( X )
- abstractmethod
-
-
-
-
-
-
-
Fits the DataProcessor to a passed DataFrame.
-Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
-Returns:
- self (DataProcessor): The fitted data processor.
-
-
- Source code in ydata_synthetic/preprocessing/base_processor.py
- @abstractmethod
-def fit ( self , X : DataFrame ) -> BaseProcessor :
- """Fits the DataProcessor to a passed DataFrame.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
- Returns:
- self (DataProcessor): The fitted data processor.
- """
-raise NotImplementedError
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Inverts the data transformation pipelines on a passed DataFrame.
-Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this DataProcessor.
- Can be used to revert transformations of training data or for synthetic samples.
-Returns:
- result (DataFrame):
- DataFrame with all performed transformations inverted.
-
-
- Source code in ydata_synthetic/preprocessing/base_processor.py
- @abstractmethod
-def inverse_transform ( self , X : ndarray ) -> DataFrame :
- """Inverts the data transformation pipelines on a passed DataFrame.
- Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this DataProcessor.
- Can be used to revert transformations of training data or for synthetic samples.
- Returns:
- result (DataFrame):
- DataFrame with all performed transformations inverted.
- """
-raise NotImplementedError
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Transforms the passed DataFrame with the fit DataProcessor.
-Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
-Returns:
- transformed (ndarray): Processed version of the passed DataFrame.
-
-
- Source code in ydata_synthetic/preprocessing/base_processor.py
- @abstractmethod
-def transform ( self , X : DataFrame ) -> ndarray :
- """Transforms the passed DataFrame with the fit DataProcessor.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
- Returns:
- transformed (ndarray): Processed version of the passed DataFrame.
- """
-raise NotImplementedError
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
CTGANDataProcessor - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- CTGANDataProcessor
-
-
-
-
-
-
-
-
- Bases: BaseProcessor
-
-
-
CTGAN data preprocessing class.
-It works like any other transformer in scikit-learn with the methods fit, transform and inverse_transform.
-Args:
- n_clusters (int), default=10:
- Number of clusters.
- epsilon (float), default=0.005:
- Epsilon value.
- num_cols (list of strings):
- List of names of numerical columns.
- cat_cols (list of strings):
- List of names of categorical columns.
-
-
- Source code in ydata_synthetic/preprocessing/regular/ctgan_processor.py
- @typechecked
-class CTGANDataProcessor ( BaseProcessor ):
- """
- CTGAN data preprocessing class.
- It works like any other transformer in scikit-learn with the methods fit, transform and inverse_transform.
- Args:
- n_clusters (int), default=10:
- Number of clusters.
- epsilon (float), default=0.005:
- Epsilon value.
- num_cols (list of strings):
- List of names of numerical columns.
- cat_cols (list of strings):
- List of names of categorical columns.
- """
-SUPPORTED_MODEL = 'CTGAN'
-def __init__ ( self , n_clusters = 10 , epsilon = 0.005 ,
-num_cols : Optional [ List [ str ]] = None ,
-cat_cols : Optional [ List [ str ]] = None ):
-super () . __init__ ( num_cols , cat_cols )
-self . _n_clusters = n_clusters
-self . _epsilon = epsilon
-self . _metadata = None
-self . _dtypes = None
-self . _output_dimensions = None
-@property
-def metadata ( self ) -> list [ ColumnMetadata ]:
- """
- Returns the metadata for each column.
- """
-return self . _metadata
-@property
-def output_dimensions ( self ) -> int :
- """
- Returns the dataset dimensionality after the preprocessing.
- """
-return int ( self . _output_dimensions )
-@ignore_warnings ( category = ConvergenceWarning )
-def fit ( self , X : pd . DataFrame ) -> CTGANDataProcessor :
- """
- Fits the data processor to a passed DataFrame.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
- Returns:
- self (CTGANDataProcessor): The fitted data processor.
- """
-self . _dtypes = X . infer_objects () . dtypes
-self . _metadata = []
-cur_idx = 0
-for column in X . columns :
-column_data = X [[ column ]] . values
-if column in self . cat_cols :
-ohe = OneHotEncoder ( sparse_output = False )
-ohe . fit ( column_data )
-n_categories = len ( ohe . categories_ [ 0 ])
-self . _metadata . append (
-ColumnMetadata (
-start_idx = cur_idx ,
-end_idx = cur_idx + n_categories ,
-discrete = True ,
-output_dim = n_categories ,
-model = ohe ,
-components = None ,
-name = column
-)
-)
-cur_idx += n_categories
-else :
-bgm = BayesianGaussianMixture (
-n_components = self . _n_clusters ,
-weight_concentration_prior_type = 'dirichlet_process' ,
-weight_concentration_prior = 0.001 ,
-n_init = 1
-)
-bgm . fit ( column_data )
-components = bgm . weights_ > self . _epsilon
-output_dim = components . sum () + 1
-self . _metadata . append (
-ColumnMetadata (
-start_idx = cur_idx ,
-end_idx = cur_idx + output_dim ,
-discrete = False ,
-output_dim = output_dim ,
-model = bgm ,
-components = components ,
-name = column
-)
-)
-cur_idx += output_dim
-self . _output_dimensions = cur_idx
-return self
-def transform ( self , X : pd . DataFrame ) -> np . ndarray :
- """
- Transforms the passed DataFrame with the fitted data processor.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
- Returns:
- Processed version of the passed DataFrame.
- """
-if self . _metadata is None :
-raise NotFittedError ( "This data processor has not yet been fitted." )
-transformed_data = []
-for col_md in self . _metadata :
-column_data = X [[ col_md . name ]] . values
-if col_md . discrete :
-ohe = col_md . model
-transformed_data . append ( ohe . transform ( column_data ))
-else :
-bgm = col_md . model
-components = col_md . components
-means = bgm . means_ . reshape (( 1 , self . _n_clusters ))
-stds = np . sqrt ( bgm . covariances_ ) . reshape (( 1 , self . _n_clusters ))
-features = ( column_data - means ) / ( 4 * stds )
-probabilities = bgm . predict_proba ( column_data )
-n_opts = components . sum ()
-features = features [:, components ]
-probabilities = probabilities [:, components ]
-opt_sel = np . zeros ( len ( column_data ), dtype = 'int' )
-for i in range ( len ( column_data )):
-norm_probs = probabilities [ i ] + 1e-6
-norm_probs = norm_probs / norm_probs . sum ()
-opt_sel [ i ] = np . random . choice ( np . arange ( n_opts ), p = norm_probs )
-idx = np . arange (( len ( features )))
-features = features [ idx , opt_sel ] . reshape ([ - 1 , 1 ])
-features = np . clip ( features , - .99 , .99 )
-probs_onehot = np . zeros_like ( probabilities )
-probs_onehot [ np . arange ( len ( probabilities )), opt_sel ] = 1
-transformed_data . append (
-np . concatenate ([ features , probs_onehot ], axis = 1 ) . astype ( float ))
-return np . concatenate ( transformed_data , axis = 1 ) . astype ( float )
-def inverse_transform ( self , X : np . ndarray ) -> pd . DataFrame :
- """
- Reverts the data transformations on a passed DataFrame.
- Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this data processor.
- Can be used to revert transformations of training data or for synthetic samples.
- Returns:
- DataFrame with all performed transformations reverted.
- """
-if self . _metadata is None :
-raise NotFittedError ( "This data processor has not yet been fitted." )
-transformed_data = []
-col_names = []
-for col_md in self . _metadata :
-col_data = X [:, col_md . start_idx : col_md . end_idx ]
-if col_md . discrete :
-inv_data = col_md . model . inverse_transform ( col_data )
-else :
-mean = col_data [:, 0 ]
-variance = col_data [:, 1 :]
-mean = np . clip ( mean , - 1 , 1 )
-v_t = np . ones (( len ( col_data ), self . _n_clusters )) * - 100
-v_t [:, col_md . components ] = variance
-variance = v_t
-means = col_md . model . means_ . reshape ([ - 1 ])
-stds = np . sqrt ( col_md . model . covariances_ ) . reshape ([ - 1 ])
-p_argmax = np . argmax ( variance , axis = 1 )
-std_t = stds [ p_argmax ]
-mean_t = means [ p_argmax ]
-inv_data = mean * 4 * std_t + mean_t
-transformed_data . append ( inv_data )
-col_names . append ( col_md . name )
-transformed_data = np . column_stack ( transformed_data )
-transformed_data = pd . DataFrame ( transformed_data , columns = col_names ) . astype ( self . _dtypes )
-return transformed_data
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Returns the metadata for each column.
-
-
-
-
-
-
-
-
-
- output_dimensions : int
- property
-
-
-
-
-
-
-
Returns the dataset dimensionality after the preprocessing.
-
-
-
-
-
-
-
-
-
-
-
-
- fit ( X )
-
-
-
-
-
Fits the data processor to a passed DataFrame.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- X
- DataFrame
-
-
-
-
DataFrame used to fit the processor parameters.
-Should be aligned with the num/cat columns defined in initialization.
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/preprocessing/regular/ctgan_processor.py
- @ignore_warnings ( category = ConvergenceWarning )
-def fit ( self , X : pd . DataFrame ) -> CTGANDataProcessor :
- """
- Fits the data processor to a passed DataFrame.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
- Returns:
- self (CTGANDataProcessor): The fitted data processor.
- """
-self . _dtypes = X . infer_objects () . dtypes
-self . _metadata = []
-cur_idx = 0
-for column in X . columns :
-column_data = X [[ column ]] . values
-if column in self . cat_cols :
-ohe = OneHotEncoder ( sparse_output = False )
-ohe . fit ( column_data )
-n_categories = len ( ohe . categories_ [ 0 ])
-self . _metadata . append (
-ColumnMetadata (
-start_idx = cur_idx ,
-end_idx = cur_idx + n_categories ,
-discrete = True ,
-output_dim = n_categories ,
-model = ohe ,
-components = None ,
-name = column
-)
-)
-cur_idx += n_categories
-else :
-bgm = BayesianGaussianMixture (
-n_components = self . _n_clusters ,
-weight_concentration_prior_type = 'dirichlet_process' ,
-weight_concentration_prior = 0.001 ,
-n_init = 1
-)
-bgm . fit ( column_data )
-components = bgm . weights_ > self . _epsilon
-output_dim = components . sum () + 1
-self . _metadata . append (
-ColumnMetadata (
-start_idx = cur_idx ,
-end_idx = cur_idx + output_dim ,
-discrete = False ,
-output_dim = output_dim ,
-model = bgm ,
-components = components ,
-name = column
-)
-)
-cur_idx += output_dim
-self . _output_dimensions = cur_idx
-return self
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Reverts the data transformations on a passed DataFrame.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- X
- ndarray
-
-
-
-
Numpy array to be brought back to the original data format.
-Should share the schema of data transformed by this data processor.
-Can be used to revert transformations of training data or for synthetic samples.
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/preprocessing/regular/ctgan_processor.py
- def inverse_transform ( self , X : np . ndarray ) -> pd . DataFrame :
- """
- Reverts the data transformations on a passed DataFrame.
- Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this data processor.
- Can be used to revert transformations of training data or for synthetic samples.
- Returns:
- DataFrame with all performed transformations reverted.
- """
-if self . _metadata is None :
-raise NotFittedError ( "This data processor has not yet been fitted." )
-transformed_data = []
-col_names = []
-for col_md in self . _metadata :
-col_data = X [:, col_md . start_idx : col_md . end_idx ]
-if col_md . discrete :
-inv_data = col_md . model . inverse_transform ( col_data )
-else :
-mean = col_data [:, 0 ]
-variance = col_data [:, 1 :]
-mean = np . clip ( mean , - 1 , 1 )
-v_t = np . ones (( len ( col_data ), self . _n_clusters )) * - 100
-v_t [:, col_md . components ] = variance
-variance = v_t
-means = col_md . model . means_ . reshape ([ - 1 ])
-stds = np . sqrt ( col_md . model . covariances_ ) . reshape ([ - 1 ])
-p_argmax = np . argmax ( variance , axis = 1 )
-std_t = stds [ p_argmax ]
-mean_t = means [ p_argmax ]
-inv_data = mean * 4 * std_t + mean_t
-transformed_data . append ( inv_data )
-col_names . append ( col_md . name )
-transformed_data = np . column_stack ( transformed_data )
-transformed_data = pd . DataFrame ( transformed_data , columns = col_names ) . astype ( self . _dtypes )
-return transformed_data
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Transforms the passed DataFrame with the fitted data processor.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- X
- DataFrame
-
-
-
-
DataFrame used to fit the processor parameters.
-Should be aligned with the columns types defined in initialization.
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/preprocessing/regular/ctgan_processor.py
- def transform ( self , X : pd . DataFrame ) -> np . ndarray :
- """
- Transforms the passed DataFrame with the fitted data processor.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
- Returns:
- Processed version of the passed DataFrame.
- """
-if self . _metadata is None :
-raise NotFittedError ( "This data processor has not yet been fitted." )
-transformed_data = []
-for col_md in self . _metadata :
-column_data = X [[ col_md . name ]] . values
-if col_md . discrete :
-ohe = col_md . model
-transformed_data . append ( ohe . transform ( column_data ))
-else :
-bgm = col_md . model
-components = col_md . components
-means = bgm . means_ . reshape (( 1 , self . _n_clusters ))
-stds = np . sqrt ( bgm . covariances_ ) . reshape (( 1 , self . _n_clusters ))
-features = ( column_data - means ) / ( 4 * stds )
-probabilities = bgm . predict_proba ( column_data )
-n_opts = components . sum ()
-features = features [:, components ]
-probabilities = probabilities [:, components ]
-opt_sel = np . zeros ( len ( column_data ), dtype = 'int' )
-for i in range ( len ( column_data )):
-norm_probs = probabilities [ i ] + 1e-6
-norm_probs = norm_probs / norm_probs . sum ()
-opt_sel [ i ] = np . random . choice ( np . arange ( n_opts ), p = norm_probs )
-idx = np . arange (( len ( features )))
-features = features [ idx , opt_sel ] . reshape ([ - 1 , 1 ])
-features = np . clip ( features , - .99 , .99 )
-probs_onehot = np . zeros_like ( probabilities )
-probs_onehot [ np . arange ( len ( probabilities )), opt_sel ] = 1
-transformed_data . append (
-np . concatenate ([ features , probs_onehot ], axis = 1 ) . astype ( float ))
-return np . concatenate ( transformed_data , axis = 1 ) . astype ( float )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
RegularDataProcessor - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- RegularDataProcessor
-
-
-
-
-
-
-
-
- Bases: BaseProcessor
-
-
-
Main class for Regular/Tabular Data Preprocessing.
-It works like any other transformer in scikit learn with the methods fit, transform and inverse transform.
-Args:
- num_cols (list of strings):
- List of names of numerical columns.
- cat_cols (list of strings):
- List of names of categorical columns.
-
-
- Source code in ydata_synthetic/preprocessing/regular/processor.py
- @typechecked
-class RegularDataProcessor ( BaseProcessor ):
- """
- Main class for Regular/Tabular Data Preprocessing.
- It works like any other transformer in scikit learn with the methods fit, transform and inverse transform.
- Args:
- num_cols (list of strings):
- List of names of numerical columns.
- cat_cols (list of strings):
- List of names of categorical columns.
- """
-def __init__ ( self , num_cols : Optional [ List [ str ]] = None , cat_cols : Optional [ List [ str ]] = None ):
-super () . __init__ ( num_cols , cat_cols )
-self . _col_order_ = None
-self . _num_col_idx_ = None
-self . _cat_col_idx_ = None
-# pylint: disable=W0106
-def fit ( self , X : DataFrame ) -> RegularDataProcessor :
- """Fits the DataProcessor to a passed DataFrame.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
- Returns:
- self (RegularDataProcessor): The fitted data processor.
- """
-self . _validate_cols ( X . columns )
-self . _col_order_ = [ c for c in X . columns if c in self . num_cols + self . cat_cols ]
-self . _types = X . dtypes
-self . _num_pipeline = Pipeline ([
-( "scaler" , MinMaxScaler ()),
-])
-self . _cat_pipeline = Pipeline ([
-( "encoder" , OneHotEncoder ( sparse_output = False , handle_unknown = 'ignore' )),
-])
-self . num_pipeline . fit ( X [ self . num_cols ]) if self . num_cols else zeros ([ len ( X ), 0 ])
-self . cat_pipeline . fit ( X [ self . cat_cols ]) if self . num_cols else zeros ([ len ( X ), 0 ])
-self . _num_col_idx_ = len ( self . num_pipeline . get_feature_names_out ())
-self . _cat_col_idx_ = self . _num_col_idx_ + len ( self . cat_pipeline . get_feature_names_out ())
-return self
-def transform ( self , X : DataFrame ) -> ndarray :
- """Transforms the passed DataFrame with the fit DataProcessor.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
- Returns:
- transformed (ndarray):
- Processed version of the passed DataFrame.
- """
-self . _check_is_fitted ()
-num_data = self . num_pipeline . transform ( X [ self . num_cols ]) if self . num_cols else zeros ([ len ( X ), 0 ])
-cat_data = self . cat_pipeline . transform ( X [ self . cat_cols ]) if self . cat_cols else zeros ([ len ( X ), 0 ])
-transformed = concatenate ([ num_data , cat_data ], axis = 1 )
-return transformed
-def inverse_transform ( self , X : ndarray ) -> DataFrame :
- """Inverts the data transformation pipelines on a passed DataFrame.
- Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this DataProcessor.
- Can be used to revert transformations of training data or for synthetic samples.
- Returns:
- result (DataFrame):
- DataFrame with all performed transformations inverted.
- """
-self . _check_is_fitted ()
-num_data , cat_data , _ = split ( X , [ self . _num_col_idx_ , self . _cat_col_idx_ ], axis = 1 )
-num_data = self . num_pipeline . inverse_transform ( num_data ) if self . num_cols else zeros ([ len ( X ), 0 ])
-cat_data = self . cat_pipeline . inverse_transform ( cat_data ) if self . cat_cols else zeros ([ len ( X ), 0 ])
-result = concat ([ DataFrame ( num_data , columns = self . num_cols ),
-DataFrame ( cat_data , columns = self . cat_cols )], axis = 1 )
-result = result . loc [:, self . _col_order_ ]
-for col in result . columns :
-result [ col ] = result [ col ] . astype ( self . _types [ col ])
-return result
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- fit ( X )
-
-
-
-
-
Fits the DataProcessor to a passed DataFrame.
-Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
-Returns:
- self (RegularDataProcessor): The fitted data processor.
-
-
- Source code in ydata_synthetic/preprocessing/regular/processor.py
- def fit ( self , X : DataFrame ) -> RegularDataProcessor :
- """Fits the DataProcessor to a passed DataFrame.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the num/cat columns defined in initialization.
- Returns:
- self (RegularDataProcessor): The fitted data processor.
- """
-self . _validate_cols ( X . columns )
-self . _col_order_ = [ c for c in X . columns if c in self . num_cols + self . cat_cols ]
-self . _types = X . dtypes
-self . _num_pipeline = Pipeline ([
-( "scaler" , MinMaxScaler ()),
-])
-self . _cat_pipeline = Pipeline ([
-( "encoder" , OneHotEncoder ( sparse_output = False , handle_unknown = 'ignore' )),
-])
-self . num_pipeline . fit ( X [ self . num_cols ]) if self . num_cols else zeros ([ len ( X ), 0 ])
-self . cat_pipeline . fit ( X [ self . cat_cols ]) if self . num_cols else zeros ([ len ( X ), 0 ])
-self . _num_col_idx_ = len ( self . num_pipeline . get_feature_names_out ())
-self . _cat_col_idx_ = self . _num_col_idx_ + len ( self . cat_pipeline . get_feature_names_out ())
-return self
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Inverts the data transformation pipelines on a passed DataFrame.
-Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this DataProcessor.
- Can be used to revert transformations of training data or for synthetic samples.
-Returns:
- result (DataFrame):
- DataFrame with all performed transformations inverted.
-
-
- Source code in ydata_synthetic/preprocessing/regular/processor.py
- def inverse_transform ( self , X : ndarray ) -> DataFrame :
- """Inverts the data transformation pipelines on a passed DataFrame.
- Args:
- X (ndarray):
- Numpy array to be brought back to the original data format.
- Should share the schema of data transformed by this DataProcessor.
- Can be used to revert transformations of training data or for synthetic samples.
- Returns:
- result (DataFrame):
- DataFrame with all performed transformations inverted.
- """
-self . _check_is_fitted ()
-num_data , cat_data , _ = split ( X , [ self . _num_col_idx_ , self . _cat_col_idx_ ], axis = 1 )
-num_data = self . num_pipeline . inverse_transform ( num_data ) if self . num_cols else zeros ([ len ( X ), 0 ])
-cat_data = self . cat_pipeline . inverse_transform ( cat_data ) if self . cat_cols else zeros ([ len ( X ), 0 ])
-result = concat ([ DataFrame ( num_data , columns = self . num_cols ),
-DataFrame ( cat_data , columns = self . cat_cols )], axis = 1 )
-result = result . loc [:, self . _col_order_ ]
-for col in result . columns :
-result [ col ] = result [ col ] . astype ( self . _types [ col ])
-return result
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Transforms the passed DataFrame with the fit DataProcessor.
-Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
-Returns:
- transformed (ndarray):
- Processed version of the passed DataFrame.
-
-
- Source code in ydata_synthetic/preprocessing/regular/processor.py
- def transform ( self , X : DataFrame ) -> ndarray :
- """Transforms the passed DataFrame with the fit DataProcessor.
- Args:
- X (DataFrame):
- DataFrame used to fit the processor parameters.
- Should be aligned with the columns types defined in initialization.
- Returns:
- transformed (ndarray):
- Processed version of the passed DataFrame.
- """
-self . _check_is_fitted ()
-num_data = self . num_pipeline . transform ( X [ self . num_cols ]) if self . num_cols else zeros ([ len ( X ), 0 ])
-cat_data = self . cat_pipeline . transform ( X [ self . cat_cols ]) if self . cat_cols else zeros ([ len ( X ), 0 ])
-transformed = concatenate ([ num_data , cat_data ], axis = 1 )
-return transformed
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Cgan - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Cgan
-
-
-
-
-
-
-
-
- Bases: ConditionalModel
-
-
-
CGAN model for discrete conditions
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- class CGAN ( ConditionalModel ):
-"CGAN model for discrete conditions"
-__MODEL__ = 'CGAN'
-def __init__ ( self , model_parameters ):
-self . _col_order = None
-super () . __init__ ( model_parameters )
-def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple]): Defaults to None
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,),
-label_shape = ( self . label_dim ),
-dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,),
-label_shape = ( self . label_dim ,),
-dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the discriminator
-self . discriminator . compile ( loss = 'binary_crossentropy' ,
-optimizer = d_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-noise = Input ( shape = ( self . noise_dim ,))
-label = Input ( shape = ( 1 ,)) # A label vector is expected
-record = self . generator ([ noise , label ])
-# For the combined model we will only train the generator
-self . discriminator . trainable = False
-# The discriminator takes generated images as input and determines validity
-validity = self . discriminator ([ record , label ])
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-self . _model = Model ([ noise , label ], validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = g_optimizer )
-def _generate_noise ( self ):
- """Gaussian noise for the generator input."""
-while True :
-yield random . uniform ( shape = ( self . noise_dim ,))
-def get_batch_noise ( self ):
- """Create a batch iterator for the generator gaussian noise input."""
-return iter ( tfdata . Dataset . from_generator ( self . _generate_noise , output_types = dtypes . float32 )
-. batch ( self . batch_size )
-. repeat ())
-def get_data_batch ( self , data , batch_size , seed = 0 ):
- """Produce real data batches from the passed data object.
- Args:
- data: real data.
- batch_size: batch size.
- seed (int, optional): Defaults to 0.
- Returns:
- data batch.
- """
-start_i = ( batch_size * seed ) % len ( data )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( data )
-np . random . seed ( shuffle_seed )
-data_ix = np . random . choice ( data . shape [ 0 ], replace = False , size = len ( data )) # wasteful to shuffle every time
-return data [ data_ix [ start_i : stop_i ]]
-def fit ( self ,
-data : DataFrame ,
-label_cols : List [ str ],
-train_arguments : TrainParameters ,
-num_cols : List [ str ],
-cat_cols : List [ str ]):
- """Trains and fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame with the data to be synthesized
- label_cols: The name of the column to be used as a label and condition for the training
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-data , label = self . _prep_fit ( data , label_cols , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . label_dim = len ( label_cols )
-# Init the GAN model and optimizers
-self . define_gan ( self . processor . col_transform_info )
-# Merging labels with processed data
-processed_data = hstack ([ processed_data , label ])
-noise_batches = self . get_batch_noise ()
-iterations = int ( abs ( processed_data . shape [ 0 ] / self . batch_size ) + 1 )
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = np . zeros (( self . batch_size , 1 ))
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_x = self . get_data_batch ( processed_data , self . batch_size ) # Batches are retrieved with labels
-batch_x , label = batch_x [:, : - 1 ], batch_x [:, - 1 ] # Separate labels from batch
-noise = next ( noise_batches )
-# Generate a batch of new records
-gen_records = self . generator ([ noise , label ], training = True )
-# Train the discriminator
-d_loss_real = self . discriminator . train_on_batch ([ batch_x , label ], valid ) # Separate labels
-d_loss_fake = self . discriminator . train_on_batch ([ gen_records , label ], fake ) # Separate labels
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = next ( noise_batches )
-# Train the generator (to have the discriminator label samples as valid)
-g_loss = self . _model . train_on_batch ([ noise , label ], valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-# If at save interval => save model state and generated image samples
-if epoch % train_arguments . sample_interval == 0 :
-self . _run_checkpoint ( train_arguments , epoch , label )
-def _run_checkpoint ( self , train_arguments , epoch , label ):
- """Run checkpoint and store model state and generated samples.
- Args:
- train_arguments: GAN training arguments.
- epoch: training epoch
- label: deprecated
- """
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . discriminator . save_weights ( model_checkpoint_base_name . format ( 'discriminator' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple]): Defaults to None
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,),
-label_shape = ( self . label_dim ),
-dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,),
-label_shape = ( self . label_dim ,),
-dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the discriminator
-self . discriminator . compile ( loss = 'binary_crossentropy' ,
-optimizer = d_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-noise = Input ( shape = ( self . noise_dim ,))
-label = Input ( shape = ( 1 ,)) # A label vector is expected
-record = self . generator ([ noise , label ])
-# For the combined model we will only train the generator
-self . discriminator . trainable = False
-# The discriminator takes generated images as input and determines validity
-validity = self . discriminator ([ record , label ])
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-self . _model = Model ([ noise , label ], validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = g_optimizer )
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , label_cols , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Trains and fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- DataFrame
-
-
-
A pandas DataFrame with the data to be synthesized
-
-
- required
-
-
-
- label_cols
- List [str]
-
-
-
The name of the column to be used as a label and condition for the training
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def fit ( self ,
-data : DataFrame ,
-label_cols : List [ str ],
-train_arguments : TrainParameters ,
-num_cols : List [ str ],
-cat_cols : List [ str ]):
- """Trains and fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame with the data to be synthesized
- label_cols: The name of the column to be used as a label and condition for the training
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-data , label = self . _prep_fit ( data , label_cols , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . label_dim = len ( label_cols )
-# Init the GAN model and optimizers
-self . define_gan ( self . processor . col_transform_info )
-# Merging labels with processed data
-processed_data = hstack ([ processed_data , label ])
-noise_batches = self . get_batch_noise ()
-iterations = int ( abs ( processed_data . shape [ 0 ] / self . batch_size ) + 1 )
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = np . zeros (( self . batch_size , 1 ))
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_x = self . get_data_batch ( processed_data , self . batch_size ) # Batches are retrieved with labels
-batch_x , label = batch_x [:, : - 1 ], batch_x [:, - 1 ] # Separate labels from batch
-noise = next ( noise_batches )
-# Generate a batch of new records
-gen_records = self . generator ([ noise , label ], training = True )
-# Train the discriminator
-d_loss_real = self . discriminator . train_on_batch ([ batch_x , label ], valid ) # Separate labels
-d_loss_fake = self . discriminator . train_on_batch ([ gen_records , label ], fake ) # Separate labels
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = next ( noise_batches )
-# Train the generator (to have the discriminator label samples as valid)
-g_loss = self . _model . train_on_batch ([ noise , label ], valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-# If at save interval => save model state and generated image samples
-if epoch % train_arguments . sample_interval == 0 :
-self . _run_checkpoint ( train_arguments , epoch , label )
-
-
-
-
-
-
-
-
-
-
-
- get_batch_noise ()
-
-
-
-
-
Create a batch iterator for the generator gaussian noise input.
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def get_batch_noise ( self ):
- """Create a batch iterator for the generator gaussian noise input."""
-return iter ( tfdata . Dataset . from_generator ( self . _generate_noise , output_types = dtypes . float32 )
-. batch ( self . batch_size )
-. repeat ())
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( data , batch_size , seed = 0 )
-
-
-
-
-
Produce real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def get_data_batch ( self , data , batch_size , seed = 0 ):
- """Produce real data batches from the passed data object.
- Args:
- data: real data.
- batch_size: batch size.
- seed (int, optional): Defaults to 0.
- Returns:
- data batch.
- """
-start_i = ( batch_size * seed ) % len ( data )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( data )
-np . random . seed ( shuffle_seed )
-data_ix = np . random . choice ( data . shape [ 0 ], replace = False , size = len ( data )) # wasteful to shuffle every time
-return data [ data_ix [ start_i : stop_i ]]
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Gan - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Gan
-
-
-
-
-
-
-
-
- Bases: BaseModel
-
-
-
Base class of GAN synthesizer models.
-The main methods are train (for fitting the synthesizer), save/load and sample (obtain synthetic records).
-Args:
- model_parameters (ModelParameters):
- Set of architectural parameters for model definition.
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- @typechecked
-class BaseGANModel ( BaseModel ):
- """
- Base class of GAN synthesizer models.
- The main methods are train (for fitting the synthesizer), save/load and sample (obtain synthetic records).
- Args:
- model_parameters (ModelParameters):
- Set of architectural parameters for model definition.
- """
-def __init__ (
-self ,
-model_parameters : ModelParameters
-):
-gpu_devices = tfconfig . list_physical_devices ( 'GPU' )
-if len ( gpu_devices ) > 0 :
-try :
-tfconfig . experimental . set_memory_growth ( gpu_devices [ 0 ], True )
-except ( ValueError , RuntimeError ):
-# Invalid device or cannot modify virtual devices once initialized.
-pass
-#Validate the provided model parameters
-if model_parameters . betas is not None :
-assert len ( model_parameters . betas ) == 2 , "Please provide the betas information as a tuple."
-self . batch_size = model_parameters . batch_size
-self . _set_lr ( model_parameters . lr )
-self . beta_1 = model_parameters . betas [ 0 ]
-self . beta_2 = model_parameters . betas [ 1 ]
-self . noise_dim = model_parameters . noise_dim
-self . data_dim = None
-self . layers_dim = model_parameters . layers_dim
-# Additional parameters for the CTGAN
-self . generator_dims = model_parameters . generator_dims
-self . critic_dims = model_parameters . critic_dims
-self . l2_scale = model_parameters . l2_scale
-self . latent_dim = model_parameters . latent_dim
-self . gp_lambda = model_parameters . gp_lambda
-self . pac = model_parameters . pac
-self . use_tanh = model_parameters . tanh
-self . processor = None
-if self . __MODEL__ in RegularModels . __members__ or \
-self . __MODEL__ == CTGANDataProcessor . SUPPORTED_MODEL :
-self . tau = model_parameters . tau_gs
-# pylint: disable=E1101
-def __call__ ( self , inputs , ** kwargs ):
-return self . model ( inputs = inputs , ** kwargs )
-# pylint: disable=C0103
-def _set_lr ( self , lr ):
-if isinstance ( lr , float ):
-self . g_lr = lr
-self . d_lr = lr
-elif isinstance ( lr ,( list , tuple )):
-assert len ( lr ) == 2 , "Please provide a two values array for the learning rates or a float."
-self . g_lr = lr [ 0 ]
-self . d_lr = lr [ 1 ]
-def define_gan ( self ):
- """Define the trainable model components.
- Optionally validate model structure with mock inputs and initialize optimizers."""
-raise NotImplementedError
-@property
-def model_parameters ( self ):
-"Returns the parameters of the model."
-return self . _model_parameters
-@property
-def model_name ( self ):
-"Returns the model (class) name."
-return self . __class__ . __name__
-def fit ( self ,
-data : Union [ DataFrame , array ],
-num_cols : Optional [ List [ str ]] = None ,
-cat_cols : Optional [ List [ str ]] = None ,
-train_arguments : Optional [ TrainParameters ] = None ) -> Union [ DataFrame , array ]:
- """
- Trains and fit a synthesizer model to a given input dataset.
- Args:
- data (Union[DataFrame, array]): Training data
- num_cols (Optional[List[str]]) : List with the names of the categorical columns
- cat_cols (Optional[List[str]]): List of names of categorical columns
- train_arguments (Optional[TrainParameters]): Training parameters
- Returns:
- Fitted synthesizer
- """
-if self . __MODEL__ in RegularModels . __members__ :
-self . processor = RegularDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ in TimeSeriesModels . __members__ :
-self . processor = TimeSeriesDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == CTGANDataProcessor . SUPPORTED_MODEL :
-n_clusters = train_arguments . n_clusters
-epsilon = train_arguments . epsilon
-self . processor = CTGANDataProcessor ( n_clusters = n_clusters , epsilon = epsilon ,
-num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == DoppelGANgerProcessor . SUPPORTED_MODEL :
-measurement_cols = train_arguments . measurement_cols
-sequence_length = train_arguments . sequence_length
-sample_length = train_arguments . sample_length
-self . processor = DoppelGANgerProcessor ( num_cols = num_cols , cat_cols = cat_cols ,
-measurement_cols = measurement_cols ,
-sequence_length = sequence_length ,
-sample_length = sample_length ,
-normalize_tanh = self . use_tanh ) . fit ( data )
-else :
-print ( f 'A DataProcessor is not available for the { self . __MODEL__ } .' )
-def sample ( self , n_samples : int ):
- """
- Generates samples from the trained synthesizer.
- Args:
- n_samples (int): Number of rows to generated.
- Returns:
- synth_sample (pandas.DataFrame): generated synthetic samples.
- """
-steps = n_samples // self . batch_size + 1
-data = []
-for _ in tqdm . trange ( steps , desc = 'Synthetic data generation' ):
-z = random . uniform ([ self . batch_size , self . noise_dim ], dtype = tf . dtypes . float32 )
-records = self . generator ( z , training = False ) . numpy ()
-data . append ( records )
-return self . processor . inverse_transform ( array ( vstack ( data )))
-def save ( self , path ):
- """
- Saves a synthesizer as a pickle.
- Args:
- path (str): Path to write the synthesizer as a pickle object.
- """
-#Save only the generator?
-if self . __MODEL__ == 'WGAN' or self . __MODEL__ == 'WGAN_GP' or self . __MODEL__ == 'CWGAN_GP' :
-del self . critic
-make_keras_picklable ()
-dump ( self , path )
-@classmethod
-def load ( cls , path ):
- """
- Loads a saved synthesizer from a pickle.
- Args:
- path (str): Path to read the synthesizer pickle from.
- """
-gpu_devices = tfconfig . list_physical_devices ( 'GPU' )
-if len ( gpu_devices ) > 0 :
-try :
-tfconfig . experimental . set_memory_growth ( gpu_devices [ 0 ], True )
-except ( ValueError , RuntimeError ):
-# Invalid device or cannot modify virtual devices once initialized.
-pass
-synth = load ( path )
-return synth
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- model_name
- property
-
-
-
-
-
-
-
Returns the model (class) name.
-
-
-
-
-
-
-
-
-
- model_parameters
- property
-
-
-
-
-
-
-
Returns the parameters of the model.
-
-
-
-
-
-
-
-
-
-
-
-
- define_gan ()
-
-
-
-
-
Define the trainable model components.
-
Optionally validate model structure with mock inputs and initialize optimizers.
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def define_gan ( self ):
- """Define the trainable model components.
- Optionally validate model structure with mock inputs and initialize optimizers."""
-raise NotImplementedError
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , num_cols = None , cat_cols = None , train_arguments = None )
-
-
-
-
-
Trains and fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- Union [DataFrame , array ]
-
-
-
-
- required
-
-
-
- num_cols
- Optional[List[str]])
-
-
-
-
List with the names of the categorical columns
-
-
- None
-
-
-
- cat_cols
- Optional [List [str]]
-
-
-
List of names of categorical columns
-
-
- None
-
-
-
- train_arguments
- Optional [TrainParameters ]
-
-
-
-
- None
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- Union [DataFrame , array ]
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def fit ( self ,
-data : Union [ DataFrame , array ],
-num_cols : Optional [ List [ str ]] = None ,
-cat_cols : Optional [ List [ str ]] = None ,
-train_arguments : Optional [ TrainParameters ] = None ) -> Union [ DataFrame , array ]:
- """
- Trains and fit a synthesizer model to a given input dataset.
- Args:
- data (Union[DataFrame, array]): Training data
- num_cols (Optional[List[str]]) : List with the names of the categorical columns
- cat_cols (Optional[List[str]]): List of names of categorical columns
- train_arguments (Optional[TrainParameters]): Training parameters
- Returns:
- Fitted synthesizer
- """
-if self . __MODEL__ in RegularModels . __members__ :
-self . processor = RegularDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ in TimeSeriesModels . __members__ :
-self . processor = TimeSeriesDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == CTGANDataProcessor . SUPPORTED_MODEL :
-n_clusters = train_arguments . n_clusters
-epsilon = train_arguments . epsilon
-self . processor = CTGANDataProcessor ( n_clusters = n_clusters , epsilon = epsilon ,
-num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == DoppelGANgerProcessor . SUPPORTED_MODEL :
-measurement_cols = train_arguments . measurement_cols
-sequence_length = train_arguments . sequence_length
-sample_length = train_arguments . sample_length
-self . processor = DoppelGANgerProcessor ( num_cols = num_cols , cat_cols = cat_cols ,
-measurement_cols = measurement_cols ,
-sequence_length = sequence_length ,
-sample_length = sample_length ,
-normalize_tanh = self . use_tanh ) . fit ( data )
-else :
-print ( f 'A DataProcessor is not available for the { self . __MODEL__ } .' )
-
-
-
-
-
-
-
-
-
-
-
- load ( path )
- classmethod
-
-
-
-
-
-
-
Loads a saved synthesizer from a pickle.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- path
- str
-
-
-
-
Path to read the synthesizer pickle from.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- @classmethod
-def load ( cls , path ):
- """
- Loads a saved synthesizer from a pickle.
- Args:
- path (str): Path to read the synthesizer pickle from.
- """
-gpu_devices = tfconfig . list_physical_devices ( 'GPU' )
-if len ( gpu_devices ) > 0 :
-try :
-tfconfig . experimental . set_memory_growth ( gpu_devices [ 0 ], True )
-except ( ValueError , RuntimeError ):
-# Invalid device or cannot modify virtual devices once initialized.
-pass
-synth = load ( path )
-return synth
-
-
-
-
-
-
-
-
-
-
-
- sample ( n_samples )
-
-
-
-
-
Generates samples from the trained synthesizer.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- n_samples
- int
-
-
-
-
Number of rows to generated.
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
-Name Type
- Description
-
-
-
-
-synth_sample
- DataFrame
-
-
-
generated synthetic samples.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def sample ( self , n_samples : int ):
- """
- Generates samples from the trained synthesizer.
- Args:
- n_samples (int): Number of rows to generated.
- Returns:
- synth_sample (pandas.DataFrame): generated synthetic samples.
- """
-steps = n_samples // self . batch_size + 1
-data = []
-for _ in tqdm . trange ( steps , desc = 'Synthetic data generation' ):
-z = random . uniform ([ self . batch_size , self . noise_dim ], dtype = tf . dtypes . float32 )
-records = self . generator ( z , training = False ) . numpy ()
-data . append ( records )
-return self . processor . inverse_transform ( array ( vstack ( data )))
-
-
-
-
-
-
-
-
-
-
-
- save ( path )
-
-
-
-
-
Saves a synthesizer as a pickle.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- path
- str
-
-
-
-
Path to write the synthesizer as a pickle object.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def save ( self , path ):
- """
- Saves a synthesizer as a pickle.
- Args:
- path (str): Path to write the synthesizer as a pickle object.
- """
-#Save only the generator?
-if self . __MODEL__ == 'WGAN' or self . __MODEL__ == 'WGAN_GP' or self . __MODEL__ == 'CWGAN_GP' :
-del self . critic
-make_keras_picklable ()
-dump ( self , path )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
CGAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- CGAN
-
-
-
-
-
-
-
-
- Bases: ConditionalModel
-
-
-
CGAN model for discrete conditions
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- class CGAN ( ConditionalModel ):
-"CGAN model for discrete conditions"
-__MODEL__ = 'CGAN'
-def __init__ ( self , model_parameters ):
-self . _col_order = None
-super () . __init__ ( model_parameters )
-def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple]): Defaults to None
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,),
-label_shape = ( self . label_dim ),
-dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,),
-label_shape = ( self . label_dim ,),
-dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the discriminator
-self . discriminator . compile ( loss = 'binary_crossentropy' ,
-optimizer = d_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-noise = Input ( shape = ( self . noise_dim ,))
-label = Input ( shape = ( 1 ,)) # A label vector is expected
-record = self . generator ([ noise , label ])
-# For the combined model we will only train the generator
-self . discriminator . trainable = False
-# The discriminator takes generated images as input and determines validity
-validity = self . discriminator ([ record , label ])
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-self . _model = Model ([ noise , label ], validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = g_optimizer )
-def _generate_noise ( self ):
- """Gaussian noise for the generator input."""
-while True :
-yield random . uniform ( shape = ( self . noise_dim ,))
-def get_batch_noise ( self ):
- """Create a batch iterator for the generator gaussian noise input."""
-return iter ( tfdata . Dataset . from_generator ( self . _generate_noise , output_types = dtypes . float32 )
-. batch ( self . batch_size )
-. repeat ())
-def get_data_batch ( self , data , batch_size , seed = 0 ):
- """Produce real data batches from the passed data object.
- Args:
- data: real data.
- batch_size: batch size.
- seed (int, optional): Defaults to 0.
- Returns:
- data batch.
- """
-start_i = ( batch_size * seed ) % len ( data )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( data )
-np . random . seed ( shuffle_seed )
-data_ix = np . random . choice ( data . shape [ 0 ], replace = False , size = len ( data )) # wasteful to shuffle every time
-return data [ data_ix [ start_i : stop_i ]]
-def fit ( self ,
-data : DataFrame ,
-label_cols : List [ str ],
-train_arguments : TrainParameters ,
-num_cols : List [ str ],
-cat_cols : List [ str ]):
- """Trains and fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame with the data to be synthesized
- label_cols: The name of the column to be used as a label and condition for the training
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-data , label = self . _prep_fit ( data , label_cols , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . label_dim = len ( label_cols )
-# Init the GAN model and optimizers
-self . define_gan ( self . processor . col_transform_info )
-# Merging labels with processed data
-processed_data = hstack ([ processed_data , label ])
-noise_batches = self . get_batch_noise ()
-iterations = int ( abs ( processed_data . shape [ 0 ] / self . batch_size ) + 1 )
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = np . zeros (( self . batch_size , 1 ))
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_x = self . get_data_batch ( processed_data , self . batch_size ) # Batches are retrieved with labels
-batch_x , label = batch_x [:, : - 1 ], batch_x [:, - 1 ] # Separate labels from batch
-noise = next ( noise_batches )
-# Generate a batch of new records
-gen_records = self . generator ([ noise , label ], training = True )
-# Train the discriminator
-d_loss_real = self . discriminator . train_on_batch ([ batch_x , label ], valid ) # Separate labels
-d_loss_fake = self . discriminator . train_on_batch ([ gen_records , label ], fake ) # Separate labels
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = next ( noise_batches )
-# Train the generator (to have the discriminator label samples as valid)
-g_loss = self . _model . train_on_batch ([ noise , label ], valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-# If at save interval => save model state and generated image samples
-if epoch % train_arguments . sample_interval == 0 :
-self . _run_checkpoint ( train_arguments , epoch , label )
-def _run_checkpoint ( self , train_arguments , epoch , label ):
- """Run checkpoint and store model state and generated samples.
- Args:
- train_arguments: GAN training arguments.
- epoch: training epoch
- label: deprecated
- """
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . discriminator . save_weights ( model_checkpoint_base_name . format ( 'discriminator' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple]): Defaults to None
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,),
-label_shape = ( self . label_dim ),
-dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,),
-label_shape = ( self . label_dim ,),
-dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the discriminator
-self . discriminator . compile ( loss = 'binary_crossentropy' ,
-optimizer = d_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-noise = Input ( shape = ( self . noise_dim ,))
-label = Input ( shape = ( 1 ,)) # A label vector is expected
-record = self . generator ([ noise , label ])
-# For the combined model we will only train the generator
-self . discriminator . trainable = False
-# The discriminator takes generated images as input and determines validity
-validity = self . discriminator ([ record , label ])
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-self . _model = Model ([ noise , label ], validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = g_optimizer )
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , label_cols , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Trains and fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- DataFrame
-
-
-
A pandas DataFrame with the data to be synthesized
-
-
- required
-
-
-
- label_cols
- List [str]
-
-
-
The name of the column to be used as a label and condition for the training
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def fit ( self ,
-data : DataFrame ,
-label_cols : List [ str ],
-train_arguments : TrainParameters ,
-num_cols : List [ str ],
-cat_cols : List [ str ]):
- """Trains and fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame with the data to be synthesized
- label_cols: The name of the column to be used as a label and condition for the training
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-data , label = self . _prep_fit ( data , label_cols , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . label_dim = len ( label_cols )
-# Init the GAN model and optimizers
-self . define_gan ( self . processor . col_transform_info )
-# Merging labels with processed data
-processed_data = hstack ([ processed_data , label ])
-noise_batches = self . get_batch_noise ()
-iterations = int ( abs ( processed_data . shape [ 0 ] / self . batch_size ) + 1 )
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = np . zeros (( self . batch_size , 1 ))
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_x = self . get_data_batch ( processed_data , self . batch_size ) # Batches are retrieved with labels
-batch_x , label = batch_x [:, : - 1 ], batch_x [:, - 1 ] # Separate labels from batch
-noise = next ( noise_batches )
-# Generate a batch of new records
-gen_records = self . generator ([ noise , label ], training = True )
-# Train the discriminator
-d_loss_real = self . discriminator . train_on_batch ([ batch_x , label ], valid ) # Separate labels
-d_loss_fake = self . discriminator . train_on_batch ([ gen_records , label ], fake ) # Separate labels
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = next ( noise_batches )
-# Train the generator (to have the discriminator label samples as valid)
-g_loss = self . _model . train_on_batch ([ noise , label ], valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-# If at save interval => save model state and generated image samples
-if epoch % train_arguments . sample_interval == 0 :
-self . _run_checkpoint ( train_arguments , epoch , label )
-
-
-
-
-
-
-
-
-
-
-
- get_batch_noise ()
-
-
-
-
-
Create a batch iterator for the generator gaussian noise input.
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def get_batch_noise ( self ):
- """Create a batch iterator for the generator gaussian noise input."""
-return iter ( tfdata . Dataset . from_generator ( self . _generate_noise , output_types = dtypes . float32 )
-. batch ( self . batch_size )
-. repeat ())
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( data , batch_size , seed = 0 )
-
-
-
-
-
Produce real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cgan/model.py
- def get_data_batch ( self , data , batch_size , seed = 0 ):
- """Produce real data batches from the passed data object.
- Args:
- data: real data.
- batch_size: batch size.
- seed (int, optional): Defaults to 0.
- Returns:
- data batch.
- """
-start_i = ( batch_size * seed ) % len ( data )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( data )
-np . random . seed ( shuffle_seed )
-data_ix = np . random . choice ( data . shape [ 0 ], replace = False , size = len ( data )) # wasteful to shuffle every time
-return data [ data_ix [ start_i : stop_i ]]
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
CRAMERGAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- CRAMERGAN
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- class CRAMERGAN ( BaseGANModel ):
-__MODEL__ = 'CRAMERGAN'
-def __init__ ( self , model_parameters , gradient_penalty_weight = 10 ):
- """Create a base CramerGAN.
- Based according to the WGAN paper - https://arxiv.org/pdf/1705.10743.pdf
- CramerGAN, a solution to biased Wassertein Gradients https://arxiv.org/abs/1705.10743"""
-self . gradient_penalty_weight = gradient_penalty_weight
-super () . __init__ ( model_parameters )
-def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-c_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# The generator takes noise as input and generates records
-z = Input ( shape = ( self . noise_dim ,), batch_size = self . batch_size )
-fake = self . generator ( z )
-logits = self . critic ( fake )
-return g_optimizer , c_optimizer
-def gradient_penalty ( self , real , fake ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- Returns:
- gradient_penalty.
- """
-gp = gradient_penalty ( self . f_crit , real , fake , mode = Mode . CRAMER )
-return gp
-def update_gradients ( self , x , g_optimizer , c_optimizer ):
- """Compute and apply the gradients for both the Generator and the Critic.
- Args:
- x: real data event
- g_optimizer: generator optimizer
- c_optimizer: critic optimizer
- Returns:
- (critic loss, generator loss)
- """
-# Update the gradients of critic for n_critic times (Training the critic)
-##New generator gradient_tape
-noise = tf . random . normal ([ x . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-noise2 = tf . random . normal ([ x . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-with tf . GradientTape () as g_tape , tf . GradientTape () as d_tape :
-fake = self . generator ( noise , training = True )
-fake2 = self . generator ( noise2 , training = True )
-g_loss = self . g_lossfn ( x , fake , fake2 )
-c_loss = self . c_lossfn ( x , fake , fake2 )
-# Get the gradients of the generator
-g_gradients = g_tape . gradient ( g_loss , self . generator . trainable_variables )
-# Update the weights of the generator
-g_optimizer . apply_gradients (
-zip ( g_gradients , self . generator . trainable_variables )
-)
-c_gradient = d_tape . gradient ( c_loss , self . critic . trainable_variables )
-# Update the weights of the critic using the optimizer
-c_optimizer . apply_gradients (
-zip ( c_gradient , self . critic . trainable_variables )
-)
-return c_loss , g_loss
-def g_lossfn ( self , real , fake , fake2 ):
- """Compute generator loss function according to the CramerGAN paper.
- Args:
- real: A real sample
- fake: A fake sample
- fak2: A second fake sample
- Returns:
- Loss of the generator
- """
-g_loss = tf . norm ( self . critic ( real , training = True ) - self . critic ( fake , training = True ), axis = 1 ) + \
-tf . norm ( self . critic ( real , training = True ) - self . critic ( fake2 , training = True ), axis = 1 ) - \
-tf . norm ( self . critic ( fake , training = True ) - self . critic ( fake2 , training = True ), axis = 1 )
-return tf . reduce_mean ( g_loss )
-def f_crit ( self , real , fake ):
- """
- Computes the critic distance function f between two samples.
- Args:
- real: A real sample
- fake: A fake sample
- Returns:
- Loss of the critic
- """
-return tf . norm ( self . critic ( real , training = True ) - self . critic ( fake , training = True ), axis = 1 ) - tf . norm ( self . critic ( real , training = True ), axis = 1 )
-def c_lossfn ( self , real , fake , fake2 ):
- """Compute the loss of the critic.
- Args:
- real: A real sample
- fake: A fake sample
- fake2: A second fake sample
- Returns:
- Loss of the critic
- """
-f_real = self . f_crit ( real , fake2 )
-f_fake = self . f_crit ( fake , fake2 )
-loss_surrogate = f_real - f_fake
-gp = self . gradient_penalty ( real , [ fake , fake2 ])
-return tf . reduce_mean ( - loss_surrogate + self . gradient_penalty_weight * gp )
-@staticmethod
-def get_data_batch ( train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-def train_step ( self , train_data , optimizers ):
- """Perform a training step.
- Args:
- train_data: training data
- optimizers: generator and critic optimizers
- Returns:
- (critic_loss, generator_loss): Critic and generator loss.
- """
-critic_loss , g_loss = self . update_gradients ( train_data , * optimizers )
-return critic_loss , g_loss
-def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ], cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-super () . fit ( data , num_cols , cat_cols )
-data = self . processor . transform ( data )
-self . data_dim = data . shape [ 1 ]
-optimizers = self . define_gan ( self . processor . col_transform_info )
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-# Create a summary file
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '..\cramergan_test' , 'summaries' , 'train' ))
-with train_summary_writer . as_default ():
-for epoch in trange ( train_arguments . epochs ):
-for iteration in range ( iterations ):
-batch_data = self . get_data_batch ( data , self . batch_size )
-c_loss , g_loss = self . train_step ( batch_data , optimizers )
-if iteration % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , iteration ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , iteration ))
-print ( f "Epoch: { epoch } | critic_loss: { c_loss } | gen_loss: { g_loss } " )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- __init__ ( model_parameters , gradient_penalty_weight = 10 )
-
-
-
-
-
Create a base CramerGAN.
-
Based according to the WGAN paper - https://arxiv.org/pdf/1705.10743.pdf
-CramerGAN, a solution to biased Wassertein Gradients https://arxiv.org/abs/1705.10743
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def __init__ ( self , model_parameters , gradient_penalty_weight = 10 ):
- """Create a base CramerGAN.
- Based according to the WGAN paper - https://arxiv.org/pdf/1705.10743.pdf
- CramerGAN, a solution to biased Wassertein Gradients https://arxiv.org/abs/1705.10743"""
-self . gradient_penalty_weight = gradient_penalty_weight
-super () . __init__ ( model_parameters )
-
-
-
-
-
-
-
-
-
-
-
- c_lossfn ( real , fake , fake2 )
-
-
-
-
-
Compute the loss of the critic.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
- fake2
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def c_lossfn ( self , real , fake , fake2 ):
- """Compute the loss of the critic.
- Args:
- real: A real sample
- fake: A fake sample
- fake2: A second fake sample
- Returns:
- Loss of the critic
- """
-f_real = self . f_crit ( real , fake2 )
-f_fake = self . f_crit ( fake , fake2 )
-loss_surrogate = f_real - f_fake
-gp = self . gradient_penalty ( real , [ fake , fake2 ])
-return tf . reduce_mean ( - loss_surrogate + self . gradient_penalty_weight * gp )
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (generator_optimizer, critic_optimizer)
-
-
-
-
Generator and critic optimizers
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-c_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# The generator takes noise as input and generates records
-z = Input ( shape = ( self . noise_dim ,), batch_size = self . batch_size )
-fake = self . generator ( z )
-logits = self . critic ( fake )
-return g_optimizer , c_optimizer
-
-
-
-
-
-
-
-
-
-
-
- f_crit ( real , fake )
-
-
-
-
-
Computes the critic distance function f between two samples.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def f_crit ( self , real , fake ):
- """
- Computes the critic distance function f between two samples.
- Args:
- real: A real sample
- fake: A fake sample
- Returns:
- Loss of the critic
- """
-return tf . norm ( self . critic ( real , training = True ) - self . critic ( fake , training = True ), axis = 1 ) - tf . norm ( self . critic ( real , training = True ), axis = 1 )
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
A pandas DataFrame or a Numpy array with the data to be synthesized
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ], cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-super () . fit ( data , num_cols , cat_cols )
-data = self . processor . transform ( data )
-self . data_dim = data . shape [ 1 ]
-optimizers = self . define_gan ( self . processor . col_transform_info )
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-# Create a summary file
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '..\cramergan_test' , 'summaries' , 'train' ))
-with train_summary_writer . as_default ():
-for epoch in trange ( train_arguments . epochs ):
-for iteration in range ( iterations ):
-batch_data = self . get_data_batch ( data , self . batch_size )
-c_loss , g_loss = self . train_step ( batch_data , optimizers )
-if iteration % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , iteration ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , iteration ))
-print ( f "Epoch: { epoch } | critic_loss: { c_loss } | gen_loss: { g_loss } " )
-
-
-
-
-
-
-
-
-
-
-
- g_lossfn ( real , fake , fake2 )
-
-
-
-
-
Compute generator loss function according to the CramerGAN paper.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
- fak2
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
Loss of the generator
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def g_lossfn ( self , real , fake , fake2 ):
- """Compute generator loss function according to the CramerGAN paper.
- Args:
- real: A real sample
- fake: A fake sample
- fak2: A second fake sample
- Returns:
- Loss of the generator
- """
-g_loss = tf . norm ( self . critic ( real , training = True ) - self . critic ( fake , training = True ), axis = 1 ) + \
-tf . norm ( self . critic ( real , training = True ) - self . critic ( fake2 , training = True ), axis = 1 ) - \
-tf . norm ( self . critic ( fake , training = True ) - self . critic ( fake2 , training = True ), axis = 1 )
-return tf . reduce_mean ( g_loss )
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( train , batch_size , seed = 0 )
- staticmethod
-
-
-
-
-
-
-
Get real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- @staticmethod
-def get_data_batch ( train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-
-
-
-
-
-
-
-
-
-
-
- gradient_penalty ( real , fake )
-
-
-
-
-
Compute gradient penalty.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def gradient_penalty ( self , real , fake ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- Returns:
- gradient_penalty.
- """
-gp = gradient_penalty ( self . f_crit , real , fake , mode = Mode . CRAMER )
-return gp
-
-
-
-
-
-
-
-
-
-
-
- train_step ( train_data , optimizers )
-
-
-
-
-
Perform a training step.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train_data
-
-
-
-
-
- required
-
-
-
- optimizers
-
-
-
-
generator and critic optimizers
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (critic_loss, generator_loss)
-
-
-
-
Critic and generator loss.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def train_step ( self , train_data , optimizers ):
- """Perform a training step.
- Args:
- train_data: training data
- optimizers: generator and critic optimizers
- Returns:
- (critic_loss, generator_loss): Critic and generator loss.
- """
-critic_loss , g_loss = self . update_gradients ( train_data , * optimizers )
-return critic_loss , g_loss
-
-
-
-
-
-
-
-
-
-
-
- update_gradients ( x , g_optimizer , c_optimizer )
-
-
-
-
-
Compute and apply the gradients for both the Generator and the Critic.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- x
-
-
-
-
-
- required
-
-
-
- g_optimizer
-
-
-
-
-
- required
-
-
-
- c_optimizer
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cramergan/model.py
- def update_gradients ( self , x , g_optimizer , c_optimizer ):
- """Compute and apply the gradients for both the Generator and the Critic.
- Args:
- x: real data event
- g_optimizer: generator optimizer
- c_optimizer: critic optimizer
- Returns:
- (critic loss, generator loss)
- """
-# Update the gradients of critic for n_critic times (Training the critic)
-##New generator gradient_tape
-noise = tf . random . normal ([ x . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-noise2 = tf . random . normal ([ x . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-with tf . GradientTape () as g_tape , tf . GradientTape () as d_tape :
-fake = self . generator ( noise , training = True )
-fake2 = self . generator ( noise2 , training = True )
-g_loss = self . g_lossfn ( x , fake , fake2 )
-c_loss = self . c_lossfn ( x , fake , fake2 )
-# Get the gradients of the generator
-g_gradients = g_tape . gradient ( g_loss , self . generator . trainable_variables )
-# Update the weights of the generator
-g_optimizer . apply_gradients (
-zip ( g_gradients , self . generator . trainable_variables )
-)
-c_gradient = d_tape . gradient ( c_loss , self . critic . trainable_variables )
-# Update the weights of the critic using the optimizer
-c_optimizer . apply_gradients (
-zip ( c_gradient , self . critic . trainable_variables )
-)
-return c_loss , g_loss
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
CTGAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- CTGAN
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
Conditional Tabular GAN model.
-Based on the paper https://arxiv.org/abs/1907.00503.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- model_parameters
- ModelParameters
-
-
-
Parameters used to create the CTGAN model.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/ctgan/model.py
- class CTGAN ( BaseGANModel ):
- """
- Conditional Tabular GAN model.
- Based on the paper https://arxiv.org/abs/1907.00503.
- Args:
- model_parameters: Parameters used to create the CTGAN model.
- """
-__MODEL__ = 'CTGAN'
-def __init__ ( self , model_parameters : ModelParameters ):
-super () . __init__ ( model_parameters )
-if self . batch_size % 2 != 0 or self . batch_size % self . pac != 0 :
-raise ValueError ( "The batch size needs to be an even value divisible by the PAC." )
-self . _model_parameters = model_parameters
-self . _real_data_sampler = None
-self . _conditional_sampler = None
-self . _generator_model = None
-self . _critic_model = None
-@staticmethod
-def _create_generator_model ( input_dim , generator_dims , data_dim , metadata , tau ):
- """
- Creates the generator model.
- Args:
- input_dim: Input dimensionality.
- generator_dims: Dimensions of each hidden layer.
- data_dim: Output dimensionality.
- metadata: Dataset columns metadata.
- tau: Gumbel-Softmax non-negative temperature.
- """
-input = Input ( shape = ( input_dim , ))
-x = input
-dim = input_dim
-for layer_dim in generator_dims :
-layer_input = x
-x = Dense ( layer_dim ,
-kernel_initializer = "random_uniform" ,
-bias_initializer = "random_uniform" )( x )
-x = BatchNormalization ( epsilon = 1e-5 , momentum = 0.9 )( x )
-x = ReLU ()( x )
-x = Concatenate ( axis = 1 )([ x , layer_input ])
-dim += layer_dim
-def _gumbel_softmax ( logits , tau = 1.0 ):
- """Applies the Gumbel-Softmax function to the given logits."""
-gumbel_dist = tfp . distributions . Gumbel ( loc = 0 , scale = 1 )
-gumbels = gumbel_dist . sample ( tf . shape ( logits ))
-gumbels = ( logits + gumbels ) / tau
-return tf . nn . softmax ( gumbels , - 1 )
-def _generator_activation ( data ):
- """Custom activation function for the generator model."""
-data_transformed = []
-for col_md in metadata :
-if col_md . discrete :
-logits = data [:, col_md . start_idx : col_md . end_idx ]
-data_transformed . append ( _gumbel_softmax ( logits , tau = tau ))
-else :
-data_transformed . append ( tf . math . tanh ( data [:, col_md . start_idx : col_md . start_idx + 1 ]))
-logits = data [:, col_md . start_idx + 1 : col_md . end_idx ]
-data_transformed . append ( _gumbel_softmax ( logits , tau = tau ))
-return data , tf . concat ( data_transformed , axis = 1 )
-x = Dense ( data_dim , kernel_initializer = "random_uniform" ,
-bias_initializer = "random_uniform" ,
-activation = _generator_activation )( x )
-return Model ( inputs = input , outputs = x )
-@staticmethod
-def _create_critic_model ( input_dim , critic_dims , pac ):
- """
- Creates the critic model.
- Args:
- input_dim: Input dimensionality.
- critic_dims: Dimensions of each hidden layer.
- pac: PAC size.
- """
-input = Input ( shape = ( input_dim ,))
-x = tf . reshape ( input , [ - 1 , input_dim * pac ])
-for dim in critic_dims :
-x = Dense ( dim ,
-kernel_initializer = "random_uniform" ,
-bias_initializer = "random_uniform" )( x )
-x = LeakyReLU ( 0.2 )( x )
-x = Dropout ( 0.5 )( x )
-x = Dense ( 1 , kernel_initializer = "random_uniform" ,
-bias_initializer = "random_uniform" )( x )
-return Model ( inputs = input , outputs = x )
-def fit ( self , data : DataFrame , train_arguments : TrainParameters , num_cols : list [ str ], cat_cols : list [ str ]):
- """
- Fits the CTGAN model.
- Args:
- data: A pandas DataFrame with the data to be synthesized.
- train_arguments: CTGAN training arguments.
- num_cols: List of columns to be handled as numerical
- cat_cols: List of columns to be handled as categorical
- """
-super () . fit ( data = data , num_cols = num_cols , cat_cols = cat_cols , train_arguments = train_arguments )
-self . _generator_optimizer = tf . keras . optimizers . Adam (
-learning_rate = self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-self . _critic_optimizer = tf . keras . optimizers . Adam (
-learning_rate = self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-train_data = self . processor . transform ( data )
-metadata = self . processor . metadata
-data_dim = self . processor . output_dimensions
-self . _real_data_sampler = RealDataSampler ( train_data , metadata )
-self . _conditional_sampler = ConditionalSampler ( train_data , metadata , train_arguments . log_frequency )
-gen_input_dim = self . latent_dim + self . _conditional_sampler . output_dimensions
-self . _generator_model = self . _create_generator_model (
-gen_input_dim , self . generator_dims , data_dim , metadata , self . tau )
-crt_input_dim = data_dim + self . _conditional_sampler . output_dimensions
-self . _critic_model = self . _create_critic_model ( crt_input_dim , self . critic_dims , self . pac )
-self . _generator_model . build (( self . batch_size , gen_input_dim ))
-self . _critic_model . build (( self . batch_size , crt_input_dim ))
-steps_per_epoch = max ( len ( train_data ) // self . batch_size , 1 )
-for epoch in range ( train_arguments . epochs ):
-for _ in range ( steps_per_epoch ):
-fake_z = tf . random . normal ([ self . batch_size , self . latent_dim ])
-cond_vector = self . _conditional_sampler . sample ( self . batch_size )
-if cond_vector is None :
-real = self . _real_data_sampler . sample ( self . batch_size )
-else :
-cond , _ , col_idx , opt_idx = cond_vector
-cond = tf . convert_to_tensor ( cond )
-fake_z = tf . concat ([ fake_z , cond ], 1 )
-perm = np . arange ( self . batch_size )
-np . random . shuffle ( perm )
-real = self . _real_data_sampler . sample_col ( col_idx [ perm ], opt_idx [ perm ])
-cond_perm = tf . gather ( cond , perm )
-fake , fake_act = self . _generator_model ( fake_z , training = True )
-real = tf . convert_to_tensor ( real . astype ( 'float32' ))
-real_cat = real if cond_vector is None else tf . concat ([ real , cond_perm ], 1 )
-fake_cat = fake if cond_vector is None else tf . concat ([ fake_act , cond ], 1 )
-critic_loss = self . _train_critic_step ( real_cat , fake_cat )
-fake_z = tf . random . normal ([ self . batch_size , self . latent_dim ])
-cond_vector = self . _conditional_sampler . sample ( self . batch_size )
-if cond_vector is None :
-generator_loss = self . _train_generator_step ( fake_z )
-else :
-cond , mask , _ , _ = cond_vector
-cond = tf . convert_to_tensor ( cond )
-mask = tf . convert_to_tensor ( mask )
-fake_z = tf . concat ([ fake_z , cond ], axis = 1 )
-generator_loss = self . _train_generator_step ( fake_z , cond , mask , metadata )
-print ( f "Epoch: { epoch } | critic_loss: { critic_loss } | generator_loss: { generator_loss } " )
-def _train_critic_step ( self , real , fake ):
- """
- Single training iteration of the critic model.
- Args:
- real: Real data.
- fake: Fake data.
- """
-with tf . GradientTape () as tape :
-y_real = self . _critic_model ( real , training = True )
-y_fake = self . _critic_model ( fake , training = True )
-gp = gradient_penalty (
-partial ( self . _critic_model , training = True ), real , fake , ModeGP . CTGAN , self . pac )
-rec_loss = - ( tf . reduce_mean ( y_real ) - tf . reduce_mean ( y_fake ))
-critic_loss = rec_loss + gp * self . gp_lambda
-gradient = tape . gradient ( critic_loss , self . _critic_model . trainable_variables )
-self . _apply_critic_gradients ( gradient , self . _critic_model . trainable_variables )
-return critic_loss
-@tf . function
-def _apply_critic_gradients ( self , gradient , trainable_variables ):
- """
- Updates gradients of the critic model.
- This logic is isolated in order to be optimized as a TF function.
- Args:
- gradient: Gradient.
- trainable_variables: Variables to be updated.
- """
-self . _critic_optimizer . apply_gradients ( zip ( gradient , trainable_variables ))
-def _train_generator_step ( self , fake_z , cond_vector = None , mask = None , metadata = None ):
- """
- Single training iteration of the generator model.
- Args:
- real: Real data.
- fake: Fake data.
- cond_vector: Conditional vector.
- mask: Mask vector.
- metadata: Dataset columns metadata.
- """
-with tf . GradientTape () as tape :
-fake , fake_act = self . _generator_model ( fake_z , training = True )
-if cond_vector is not None :
-y_fake = self . _critic_model (
-tf . concat ([ fake_act , cond_vector ], 1 ), training = True )
-cond_loss = ConditionalLoss . compute ( fake , cond_vector , mask , metadata )
-generator_loss = - tf . reduce_mean ( y_fake ) + cond_loss
-else :
-y_fake = self . _critic_model ( fake_act , training = True )
-generator_loss = - tf . reduce_mean ( y_fake )
-gradient = tape . gradient ( generator_loss , self . _generator_model . trainable_variables )
-gradient = [ gradient [ i ] + self . l2_scale * self . _generator_model . trainable_variables [ i ] for i in range ( len ( gradient ))]
-self . _apply_generator_gradients ( gradient , self . _generator_model . trainable_variables )
-return generator_loss
-@tf . function
-def _apply_generator_gradients ( self , gradient , trainable_variables ):
- """
- Updates gradients of the generator model.
- This logic is isolated in order to be optimized as a TF function.
- Args:
- gradient: Gradient.
- trainable_variables: Variables to be updated.
- """
-self . _generator_optimizer . apply_gradients ( zip ( gradient , trainable_variables ))
-def sample ( self , n_samples : int ):
- """
- Samples new data from the CTGAN.
- Args:
- n_samples: Number of samples to be generated.
- """
-if n_samples <= 0 :
-raise ValueError ( "Invalid number of samples." )
-steps = n_samples // self . batch_size + 1
-data = []
-for _ in tf . range ( steps ):
-fake_z = tf . random . normal ([ self . batch_size , self . latent_dim ])
-cond_vec = self . _conditional_sampler . sample ( self . batch_size , from_active_bits = True )
-if cond_vec is not None :
-cond = tf . constant ( cond_vec )
-fake_z = tf . concat ([ fake_z , cond ], 1 )
-fake = self . _generator_model ( fake_z )[ 1 ]
-data . append ( fake . numpy ())
-data = np . concatenate ( data , 0 )
-data = data [: n_samples ]
-return self . processor . inverse_transform ( data )
-def save ( self , path ):
- """
- Save the CTGAN model in a pickle file.
- Only the required components to sample new data are saved.
- Args:
- path: Path of the pickle file.
- """
-dump ({
-"model_parameters" : self . _model_parameters ,
-"data_dim" : self . processor . output_dimensions ,
-"gen_input_dim" : self . latent_dim + self . _conditional_sampler . output_dimensions ,
-"generator_dims" : self . generator_dims ,
-"tau" : self . tau ,
-"metadata" : self . processor . metadata ,
-"batch_size" : self . batch_size ,
-"latent_dim" : self . latent_dim ,
-"conditional_sampler" : self . _conditional_sampler . __dict__ ,
-"generator_model_weights" : self . _generator_model . get_weights (),
-"processor" : self . processor . __dict__
-}, path )
-@staticmethod
-def load ( class_dict ):
- """
- Load the CTGAN model from a pickle file.
- Only the required components to sample new data are loaded.
- Args:
- class_dict: Class dict loaded from the pickle file.
- """
-new_instance = CTGAN ( class_dict [ "model_parameters" ])
-setattr ( new_instance , "generator_dims" , class_dict [ "generator_dims" ])
-setattr ( new_instance , "tau" , class_dict [ "tau" ])
-setattr ( new_instance , "batch_size" , class_dict [ "batch_size" ])
-setattr ( new_instance , "latent_dim" , class_dict [ "latent_dim" ])
-new_instance . _conditional_sampler = ConditionalSampler ()
-new_instance . _conditional_sampler . __dict__ = class_dict [ "conditional_sampler" ]
-new_instance . processor = CTGANDataProcessor ()
-new_instance . processor . __dict__ = class_dict [ "processor" ]
-new_instance . _generator_model = new_instance . _create_generator_model (
-class_dict [ "gen_input_dim" ], class_dict [ "generator_dims" ],
-class_dict [ "data_dim" ], class_dict [ "metadata" ], class_dict [ "tau" ])
-new_instance . _generator_model . build (( class_dict [ "batch_size" ], class_dict [ "gen_input_dim" ]))
-new_instance . _generator_model . set_weights ( class_dict [ 'generator_model_weights' ])
-return new_instance
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Fits the CTGAN model.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- DataFrame
-
-
-
A pandas DataFrame with the data to be synthesized.
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
CTGAN training arguments.
-
-
- required
-
-
-
- num_cols
- list[str]
-
-
-
-
List of columns to be handled as numerical
-
-
- required
-
-
-
- cat_cols
- list[str]
-
-
-
-
List of columns to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/ctgan/model.py
- def fit ( self , data : DataFrame , train_arguments : TrainParameters , num_cols : list [ str ], cat_cols : list [ str ]):
- """
- Fits the CTGAN model.
- Args:
- data: A pandas DataFrame with the data to be synthesized.
- train_arguments: CTGAN training arguments.
- num_cols: List of columns to be handled as numerical
- cat_cols: List of columns to be handled as categorical
- """
-super () . fit ( data = data , num_cols = num_cols , cat_cols = cat_cols , train_arguments = train_arguments )
-self . _generator_optimizer = tf . keras . optimizers . Adam (
-learning_rate = self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-self . _critic_optimizer = tf . keras . optimizers . Adam (
-learning_rate = self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-train_data = self . processor . transform ( data )
-metadata = self . processor . metadata
-data_dim = self . processor . output_dimensions
-self . _real_data_sampler = RealDataSampler ( train_data , metadata )
-self . _conditional_sampler = ConditionalSampler ( train_data , metadata , train_arguments . log_frequency )
-gen_input_dim = self . latent_dim + self . _conditional_sampler . output_dimensions
-self . _generator_model = self . _create_generator_model (
-gen_input_dim , self . generator_dims , data_dim , metadata , self . tau )
-crt_input_dim = data_dim + self . _conditional_sampler . output_dimensions
-self . _critic_model = self . _create_critic_model ( crt_input_dim , self . critic_dims , self . pac )
-self . _generator_model . build (( self . batch_size , gen_input_dim ))
-self . _critic_model . build (( self . batch_size , crt_input_dim ))
-steps_per_epoch = max ( len ( train_data ) // self . batch_size , 1 )
-for epoch in range ( train_arguments . epochs ):
-for _ in range ( steps_per_epoch ):
-fake_z = tf . random . normal ([ self . batch_size , self . latent_dim ])
-cond_vector = self . _conditional_sampler . sample ( self . batch_size )
-if cond_vector is None :
-real = self . _real_data_sampler . sample ( self . batch_size )
-else :
-cond , _ , col_idx , opt_idx = cond_vector
-cond = tf . convert_to_tensor ( cond )
-fake_z = tf . concat ([ fake_z , cond ], 1 )
-perm = np . arange ( self . batch_size )
-np . random . shuffle ( perm )
-real = self . _real_data_sampler . sample_col ( col_idx [ perm ], opt_idx [ perm ])
-cond_perm = tf . gather ( cond , perm )
-fake , fake_act = self . _generator_model ( fake_z , training = True )
-real = tf . convert_to_tensor ( real . astype ( 'float32' ))
-real_cat = real if cond_vector is None else tf . concat ([ real , cond_perm ], 1 )
-fake_cat = fake if cond_vector is None else tf . concat ([ fake_act , cond ], 1 )
-critic_loss = self . _train_critic_step ( real_cat , fake_cat )
-fake_z = tf . random . normal ([ self . batch_size , self . latent_dim ])
-cond_vector = self . _conditional_sampler . sample ( self . batch_size )
-if cond_vector is None :
-generator_loss = self . _train_generator_step ( fake_z )
-else :
-cond , mask , _ , _ = cond_vector
-cond = tf . convert_to_tensor ( cond )
-mask = tf . convert_to_tensor ( mask )
-fake_z = tf . concat ([ fake_z , cond ], axis = 1 )
-generator_loss = self . _train_generator_step ( fake_z , cond , mask , metadata )
-print ( f "Epoch: { epoch } | critic_loss: { critic_loss } | generator_loss: { generator_loss } " )
-
-
-
-
-
-
-
-
-
-
-
- load ( class_dict )
- staticmethod
-
-
-
-
-
-
-
Load the CTGAN model from a pickle file.
-Only the required components to sample new data are loaded.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- class_dict
-
-
-
-
Class dict loaded from the pickle file.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/ctgan/model.py
- @staticmethod
-def load ( class_dict ):
- """
- Load the CTGAN model from a pickle file.
- Only the required components to sample new data are loaded.
- Args:
- class_dict: Class dict loaded from the pickle file.
- """
-new_instance = CTGAN ( class_dict [ "model_parameters" ])
-setattr ( new_instance , "generator_dims" , class_dict [ "generator_dims" ])
-setattr ( new_instance , "tau" , class_dict [ "tau" ])
-setattr ( new_instance , "batch_size" , class_dict [ "batch_size" ])
-setattr ( new_instance , "latent_dim" , class_dict [ "latent_dim" ])
-new_instance . _conditional_sampler = ConditionalSampler ()
-new_instance . _conditional_sampler . __dict__ = class_dict [ "conditional_sampler" ]
-new_instance . processor = CTGANDataProcessor ()
-new_instance . processor . __dict__ = class_dict [ "processor" ]
-new_instance . _generator_model = new_instance . _create_generator_model (
-class_dict [ "gen_input_dim" ], class_dict [ "generator_dims" ],
-class_dict [ "data_dim" ], class_dict [ "metadata" ], class_dict [ "tau" ])
-new_instance . _generator_model . build (( class_dict [ "batch_size" ], class_dict [ "gen_input_dim" ]))
-new_instance . _generator_model . set_weights ( class_dict [ 'generator_model_weights' ])
-return new_instance
-
-
-
-
-
-
-
-
-
-
-
- sample ( n_samples )
-
-
-
-
-
Samples new data from the CTGAN.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- n_samples
- int
-
-
-
-
Number of samples to be generated.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/ctgan/model.py
- def sample ( self , n_samples : int ):
- """
- Samples new data from the CTGAN.
- Args:
- n_samples: Number of samples to be generated.
- """
-if n_samples <= 0 :
-raise ValueError ( "Invalid number of samples." )
-steps = n_samples // self . batch_size + 1
-data = []
-for _ in tf . range ( steps ):
-fake_z = tf . random . normal ([ self . batch_size , self . latent_dim ])
-cond_vec = self . _conditional_sampler . sample ( self . batch_size , from_active_bits = True )
-if cond_vec is not None :
-cond = tf . constant ( cond_vec )
-fake_z = tf . concat ([ fake_z , cond ], 1 )
-fake = self . _generator_model ( fake_z )[ 1 ]
-data . append ( fake . numpy ())
-data = np . concatenate ( data , 0 )
-data = data [: n_samples ]
-return self . processor . inverse_transform ( data )
-
-
-
-
-
-
-
-
-
-
-
- save ( path )
-
-
-
-
-
Save the CTGAN model in a pickle file.
-Only the required components to sample new data are saved.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- path
-
-
-
-
Path of the pickle file.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/ctgan/model.py
- def save ( self , path ):
- """
- Save the CTGAN model in a pickle file.
- Only the required components to sample new data are saved.
- Args:
- path: Path of the pickle file.
- """
-dump ({
-"model_parameters" : self . _model_parameters ,
-"data_dim" : self . processor . output_dimensions ,
-"gen_input_dim" : self . latent_dim + self . _conditional_sampler . output_dimensions ,
-"generator_dims" : self . generator_dims ,
-"tau" : self . tau ,
-"metadata" : self . processor . metadata ,
-"batch_size" : self . batch_size ,
-"latent_dim" : self . latent_dim ,
-"conditional_sampler" : self . _conditional_sampler . __dict__ ,
-"generator_model_weights" : self . _generator_model . get_weights (),
-"processor" : self . processor . __dict__
-}, path )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
CWGANGP - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- CWGANGP
-
-
-
-
-
-
-
-
- Bases: ConditionalModel WGAN_GP
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- class CWGANGP ( ConditionalModel , WGAN_GP ):
-__MODEL__ = 'CWGAN_GP'
-def __init__ ( self , model_parameters ,
-n_generator : Optional [ int ] = 1 ,
-n_critic : Optional [ int ] = 1 ,
-gradient_penalty_weight : int = 10 ):
- """
- Adapts the WGAN_GP synthesizer implementation to be conditional.
- Several conditional WGAN implementations can be found online, here are a few:
- https://cameronfabbri.github.io/papers/conditionalWGAN.pdf
- https://www.sciencedirect.com/science/article/abs/pii/S0020025519309715
- https://arxiv.org/pdf/2008.09202.pdf
- """
-WGAN_GP . __init__ ( self , model_parameters ,
-n_generator = n_generator ,
-n_critic = n_critic ,
-gradient_penalty_weight = gradient_penalty_weight )
-def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple]): Defaults to None
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,),
-label_shape = ( self . label_dim , ),
-dim = self . layers_dim ,
-data_dim = self . data_dim ,
-activation_info = activation_info ,
-tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,),
-label_shape = ( self . label_dim ,),
-dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-c_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-return g_optimizer , c_optimizer
-def gradient_penalty ( self , real , fake , label ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- label: ground truth.
- Returns:
- gradient_penalty
- """
-epsilon = random . uniform ([ real . shape [ 0 ], 1 ], 0.0 , 1.0 , dtype = dtypes . float32 )
-x_hat = epsilon * real + ( 1 - epsilon ) * fake
-with GradientTape () as t :
-t . watch ( x_hat )
-d_hat = self . critic ([ x_hat , label ])
-gradients = t . gradient ( d_hat , x_hat )
-ddx = sqrt ( reduce_sum ( gradients ** 2 ))
-d_regularizer = reduce_mean (( ddx - 1.0 ) ** 2 )
-return d_regularizer
-@staticmethod
-def get_data_batch ( data , batch_size , seed = 0 ):
- """Produce real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-start_i = ( batch_size * seed ) % len ( data )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( data )
-np . random . seed ( shuffle_seed )
-data_ix = np . random . choice ( data . shape [ 0 ], replace = False , size = len ( data )) # wasteful to shuffle every time
-return dtypes . cast ( data [ data_ix [ start_i : stop_i ]], dtype = dtypes . float32 )
-def c_lossfn ( self , real ):
- """Compute the critic loss.
- Args:
- real: A real sample
- Returns:
- Critic loss
- """
-real , label = real
-# generating noise from a uniform distribution
-noise = random . uniform ([ real . shape [ 0 ], self . noise_dim ], minval = 0.999 , maxval = 1.0 , dtype = dtypes . float32 )
-# run noise through generator
-fake = self . generator ([ noise , label ])
-# discriminate x and x_gen
-logits_real = self . critic ([ real , label ])
-logits_fake = self . critic ([ fake , label ])
-# gradient penalty
-gp = self . gradient_penalty ( real , fake , label )
-# getting the loss of the critic.
-c_loss = ( reduce_mean ( logits_fake )
-- reduce_mean ( logits_real )
-+ gp * self . gradient_penalty_weight )
-return c_loss
-def g_lossfn ( self , real ):
- """
- Forward pass on the generator and computes the loss.
- Args:
- real: Data batch we are analyzing
- Returns:
- Generator loss
- """
-real , label = real
-# generating noise from a uniform distribution
-noise = random . uniform ([ real . shape [ 0 ], self . noise_dim ], minval = 0.0 , maxval = 0.001 , dtype = dtypes . float32 )
-fake = self . generator ([ noise , label ])
-logits_fake = self . critic ([ fake , label ])
-g_loss = - reduce_mean ( logits_fake )
-return g_loss
-def fit ( self , data : DataFrame ,
-label_cols : List [ str ],
-train_arguments : TrainParameters ,
-num_cols : List [ str ],
-cat_cols : List [ str ]):
- """
- Train the synthesizer on a provided dataset based on a specified condition column.
- Args:
- data: A pandas DataFrame with the data to be synthesized
- label: The name of the column to be used as a label and condition for the training
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-data , label = self . _prep_fit ( data , label_cols , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . label_dim = len ( label_cols )
-#Init the GAN model and optimizers
-optimizers = self . define_gan ( self . processor . col_transform_info )
-# Merging labels with processed data
-processed_data = hstack ([ processed_data , label ])
-iterations = int ( abs ( processed_data . shape [ 0 ] / self . batch_size ) + 1 )
-print ( f 'Number of iterations per epoch: { iterations } ' )
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_x = self . get_data_batch ( processed_data , self . batch_size ) # Batches are retrieved with labels
-batch_x , label = batch_x [:, : - self . label_dim ], batch_x [:, - self . label_dim :] # Separate labels from batch
-cri_loss , ge_loss = self . train_step (( batch_x , label ), optimizers )
-print (
-"Epoch: {} | critic_loss: {} | gen_loss: {} " . format (
-epoch , cri_loss , ge_loss
-))
-# If at save interval => save model state and generated image samples
-if epoch % train_arguments . sample_interval == 0 :
-self . _run_checkpoint ( train_arguments , epoch )
-def _run_checkpoint ( self , train_arguments , epoch ):
-"Run checkpoint and store model state and generated samples."
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- __init__ ( model_parameters , n_generator = 1 , n_critic = 1 , gradient_penalty_weight = 10 )
-
-
-
-
-
Adapts the WGAN_GP synthesizer implementation to be conditional.
-
Several conditional WGAN implementations can be found online, here are a few:
- https://cameronfabbri.github.io/papers/conditionalWGAN.pdf
- https://www.sciencedirect.com/science/article/abs/pii/S0020025519309715
- https://arxiv.org/pdf/2008.09202.pdf
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- def __init__ ( self , model_parameters ,
-n_generator : Optional [ int ] = 1 ,
-n_critic : Optional [ int ] = 1 ,
-gradient_penalty_weight : int = 10 ):
- """
- Adapts the WGAN_GP synthesizer implementation to be conditional.
- Several conditional WGAN implementations can be found online, here are a few:
- https://cameronfabbri.github.io/papers/conditionalWGAN.pdf
- https://www.sciencedirect.com/science/article/abs/pii/S0020025519309715
- https://arxiv.org/pdf/2008.09202.pdf
- """
-WGAN_GP . __init__ ( self , model_parameters ,
-n_generator = n_generator ,
-n_critic = n_critic ,
-gradient_penalty_weight = gradient_penalty_weight )
-
-
-
-
-
-
-
-
-
-
-
- c_lossfn ( real )
-
-
-
-
-
Compute the critic loss.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- def c_lossfn ( self , real ):
- """Compute the critic loss.
- Args:
- real: A real sample
- Returns:
- Critic loss
- """
-real , label = real
-# generating noise from a uniform distribution
-noise = random . uniform ([ real . shape [ 0 ], self . noise_dim ], minval = 0.999 , maxval = 1.0 , dtype = dtypes . float32 )
-# run noise through generator
-fake = self . generator ([ noise , label ])
-# discriminate x and x_gen
-logits_real = self . critic ([ real , label ])
-logits_fake = self . critic ([ fake , label ])
-# gradient penalty
-gp = self . gradient_penalty ( real , fake , label )
-# getting the loss of the critic.
-c_loss = ( reduce_mean ( logits_fake )
-- reduce_mean ( logits_real )
-+ gp * self . gradient_penalty_weight )
-return c_loss
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple]): Defaults to None
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,),
-label_shape = ( self . label_dim , ),
-dim = self . layers_dim ,
-data_dim = self . data_dim ,
-activation_info = activation_info ,
-tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,),
-label_shape = ( self . label_dim ,),
-dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-c_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-return g_optimizer , c_optimizer
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , label_cols , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Train the synthesizer on a provided dataset based on a specified condition column.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- DataFrame
-
-
-
A pandas DataFrame with the data to be synthesized
-
-
- required
-
-
-
- label
-
-
-
-
The name of the column to be used as a label and condition for the training
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- def fit ( self , data : DataFrame ,
-label_cols : List [ str ],
-train_arguments : TrainParameters ,
-num_cols : List [ str ],
-cat_cols : List [ str ]):
- """
- Train the synthesizer on a provided dataset based on a specified condition column.
- Args:
- data: A pandas DataFrame with the data to be synthesized
- label: The name of the column to be used as a label and condition for the training
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-data , label = self . _prep_fit ( data , label_cols , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . label_dim = len ( label_cols )
-#Init the GAN model and optimizers
-optimizers = self . define_gan ( self . processor . col_transform_info )
-# Merging labels with processed data
-processed_data = hstack ([ processed_data , label ])
-iterations = int ( abs ( processed_data . shape [ 0 ] / self . batch_size ) + 1 )
-print ( f 'Number of iterations per epoch: { iterations } ' )
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_x = self . get_data_batch ( processed_data , self . batch_size ) # Batches are retrieved with labels
-batch_x , label = batch_x [:, : - self . label_dim ], batch_x [:, - self . label_dim :] # Separate labels from batch
-cri_loss , ge_loss = self . train_step (( batch_x , label ), optimizers )
-print (
-"Epoch: {} | critic_loss: {} | gen_loss: {} " . format (
-epoch , cri_loss , ge_loss
-))
-# If at save interval => save model state and generated image samples
-if epoch % train_arguments . sample_interval == 0 :
-self . _run_checkpoint ( train_arguments , epoch )
-
-
-
-
-
-
-
-
-
-
-
- g_lossfn ( real )
-
-
-
-
-
Forward pass on the generator and computes the loss.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
Data batch we are analyzing
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- def g_lossfn ( self , real ):
- """
- Forward pass on the generator and computes the loss.
- Args:
- real: Data batch we are analyzing
- Returns:
- Generator loss
- """
-real , label = real
-# generating noise from a uniform distribution
-noise = random . uniform ([ real . shape [ 0 ], self . noise_dim ], minval = 0.0 , maxval = 0.001 , dtype = dtypes . float32 )
-fake = self . generator ([ noise , label ])
-logits_fake = self . critic ([ fake , label ])
-g_loss = - reduce_mean ( logits_fake )
-return g_loss
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( data , batch_size , seed = 0 )
- staticmethod
-
-
-
-
-
-
-
Produce real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- @staticmethod
-def get_data_batch ( data , batch_size , seed = 0 ):
- """Produce real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-start_i = ( batch_size * seed ) % len ( data )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( data )
-np . random . seed ( shuffle_seed )
-data_ix = np . random . choice ( data . shape [ 0 ], replace = False , size = len ( data )) # wasteful to shuffle every time
-return dtypes . cast ( data [ data_ix [ start_i : stop_i ]], dtype = dtypes . float32 )
-
-
-
-
-
-
-
-
-
-
-
- gradient_penalty ( real , fake , label )
-
-
-
-
-
Compute gradient penalty.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
- label
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/cwgangp/model.py
- def gradient_penalty ( self , real , fake , label ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- label: ground truth.
- Returns:
- gradient_penalty
- """
-epsilon = random . uniform ([ real . shape [ 0 ], 1 ], 0.0 , 1.0 , dtype = dtypes . float32 )
-x_hat = epsilon * real + ( 1 - epsilon ) * fake
-with GradientTape () as t :
-t . watch ( x_hat )
-d_hat = self . critic ([ x_hat , label ])
-gradients = t . gradient ( d_hat , x_hat )
-ddx = sqrt ( reduce_sum ( gradients ** 2 ))
-d_regularizer = reduce_mean (( ddx - 1.0 ) ** 2 )
-return d_regularizer
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
DRAGAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- DRAGAN
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- class DRAGAN ( BaseGANModel ):
-__MODEL__ = 'DRAGAN'
-def __init__ ( self , model_parameters , n_discriminator , gradient_penalty_weight = 10 ):
- """DRAGAN model architecture implementation.
- Args:
- model_parameters:
- n_discriminator:
- gradient_penalty_weight (int, optional): Defaults to 10.
- """
-# As recommended in DRAGAN paper - https://arxiv.org/abs/1705.07215
-self . n_discriminator = n_discriminator
-self . gradient_penalty_weight = gradient_penalty_weight
-super () . __init__ ( model_parameters )
-def define_gan ( self , col_transform_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- col_transform_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, discriminator_optimizer): Generator and discriminator optimizers
- """
-# define generator/discriminator
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = col_transform_info , tau = self . tau )
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 , clipvalue = 0.001 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 , clipvalue = 0.001 )
-return g_optimizer , d_optimizer
-def gradient_penalty ( self , real , fake ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- Returns:
- gradient_penalty.
- """
-gp = gradient_penalty ( self . discriminator , real , fake , mode = Mode . DRAGAN )
-return gp
-def update_gradients ( self , x , g_optimizer , d_optimizer ):
- """Compute the gradients for Generator and Discriminator.
- Args:
- x (tf.tensor): real data event
- g_optimizer (tf.OptimizerV2): Optimizer for the generator model
- c_optimizer (tf.OptimizerV2): Optimizer for the discriminator model
- Returns:
- (discriminator loss, generator loss)
- """
-# Update the gradients of critic for n_critic times (Training the critic)
-for _ in range ( self . n_discriminator ):
-with tf . GradientTape () as d_tape :
-d_loss = self . d_lossfn ( x )
-# Get the gradients of the critic
-d_gradient = d_tape . gradient ( d_loss , self . discriminator . trainable_variables )
-# Update the weights of the critic using the optimizer
-d_optimizer . apply_gradients (
-zip ( d_gradient , self . discriminator . trainable_variables )
-)
-# Update the generator
-with tf . GradientTape () as g_tape :
-gen_loss = self . g_lossfn ( x )
-# Get the gradients of the generator
-gen_gradients = g_tape . gradient ( gen_loss , self . generator . trainable_variables )
-# Update the weights of the generator
-g_optimizer . apply_gradients (
-zip ( gen_gradients , self . generator . trainable_variables )
-)
-return d_loss , gen_loss
-def d_lossfn ( self , real ):
- """Calculates the critic losses.
- Args:
- real: real data examples.
- Returns:
- discriminator loss
- """
-noise = tf . random . normal (( self . batch_size , self . noise_dim ), dtype = tf . dtypes . float64 )
-# run noise through generator
-fake = self . generator ( noise )
-# discriminate x and x_gen
-logits_real = self . discriminator ( real , training = True )
-logits_fake = self . discriminator ( fake , training = True )
-# gradient penalty
-gp = self . gradient_penalty ( real , fake )
-# getting the loss of the discriminator.
-d_loss = ( tf . reduce_mean ( logits_fake )
-- tf . reduce_mean ( logits_real )
-+ gp * self . gradient_penalty_weight )
-return d_loss
-def g_lossfn ( self , real ):
- """Calculates the Generator losses.
- Args:
- real: real data.
- Returns:
- generator loss
- """
-# generating noise from a uniform distribution
-noise = tf . random . normal (( real . shape [ 0 ], self . noise_dim ), dtype = tf . float64 )
-fake = self . generator ( noise , training = True )
-logits_fake = self . discriminator ( fake , training = True )
-g_loss = - tf . reduce_mean ( logits_fake )
-return g_loss
-def get_data_batch ( self , train , batch_size ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-buffer_size = len ( train )
-#tensor_data = pd.concat([x_train, y_train], axis=1)
-train_loader = tf . data . Dataset . from_tensor_slices ( train ) \
-. batch ( batch_size ) . shuffle ( buffer_size )
-return train_loader
-def train_step ( self , train_data , optimizers ):
- """Perform a training step.
- Args:
- train_data: training data
- optimizers: generator and critic optimizers
- Returns:
- (critic_loss, generator_loss): Critic and generator loss.
- """
-d_loss , g_loss = self . update_gradients ( train_data , * optimizers )
-return d_loss , g_loss
-def fit ( self , data , train_arguments , num_cols , cat_cols ):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-optimizers = self . define_gan ( self . processor . col_transform_info )
-train_loader = self . get_data_batch ( processed_data , self . batch_size )
-# Create a summary file
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '..\dragan_test' , 'summaries' , 'train' ))
-with train_summary_writer . as_default ():
-for epoch in tqdm . trange ( train_arguments . epochs ):
-for batch_data in train_loader :
-batch_data = tf . cast ( batch_data , dtype = tf . float32 )
-d_loss , g_loss = self . train_step ( batch_data , optimizers )
-print (
-"Epoch: {} | disc_loss: {} | gen_loss: {} " . format (
-epoch , d_loss , g_loss
-))
-if epoch % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . discriminator . save_weights ( model_checkpoint_base_name . format ( 'discriminator' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- __init__ ( model_parameters , n_discriminator , gradient_penalty_weight = 10 )
-
-
-
-
-
DRAGAN model architecture implementation.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- model_parameters
-
-
-
-
-
-
-
- required
-
-
-
- n_discriminator
-
-
-
-
-
-
-
- required
-
-
-
- gradient_penalty_weight
- int
-
-
-
-
-
- 10
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def __init__ ( self , model_parameters , n_discriminator , gradient_penalty_weight = 10 ):
- """DRAGAN model architecture implementation.
- Args:
- model_parameters:
- n_discriminator:
- gradient_penalty_weight (int, optional): Defaults to 10.
- """
-# As recommended in DRAGAN paper - https://arxiv.org/abs/1705.07215
-self . n_discriminator = n_discriminator
-self . gradient_penalty_weight = gradient_penalty_weight
-super () . __init__ ( model_parameters )
-
-
-
-
-
-
-
-
-
-
-
- d_lossfn ( real )
-
-
-
-
-
Calculates the critic losses.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def d_lossfn ( self , real ):
- """Calculates the critic losses.
- Args:
- real: real data examples.
- Returns:
- discriminator loss
- """
-noise = tf . random . normal (( self . batch_size , self . noise_dim ), dtype = tf . dtypes . float64 )
-# run noise through generator
-fake = self . generator ( noise )
-# discriminate x and x_gen
-logits_real = self . discriminator ( real , training = True )
-logits_fake = self . discriminator ( fake , training = True )
-# gradient penalty
-gp = self . gradient_penalty ( real , fake )
-# getting the loss of the discriminator.
-d_loss = ( tf . reduce_mean ( logits_fake )
-- tf . reduce_mean ( logits_real )
-+ gp * self . gradient_penalty_weight )
-return d_loss
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( col_transform_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- col_transform_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (generator_optimizer, discriminator_optimizer)
-
-
-
-
Generator and discriminator optimizers
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def define_gan ( self , col_transform_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- col_transform_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, discriminator_optimizer): Generator and discriminator optimizers
- """
-# define generator/discriminator
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = col_transform_info , tau = self . tau )
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 , clipvalue = 0.001 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 , clipvalue = 0.001 )
-return g_optimizer , d_optimizer
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
A pandas DataFrame or a Numpy array with the data to be synthesized
-
-
- required
-
-
-
- train_arguments
-
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
-
-
-
-
List of columns of the data object to be handled as numerical
-
-
- required
-
-
-
- cat_cols
-
-
-
-
List of columns of the data object to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def fit ( self , data , train_arguments , num_cols , cat_cols ):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized
- train_arguments: GAN training arguments.
- num_cols: List of columns of the data object to be handled as numerical
- cat_cols: List of columns of the data object to be handled as categorical
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-optimizers = self . define_gan ( self . processor . col_transform_info )
-train_loader = self . get_data_batch ( processed_data , self . batch_size )
-# Create a summary file
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '..\dragan_test' , 'summaries' , 'train' ))
-with train_summary_writer . as_default ():
-for epoch in tqdm . trange ( train_arguments . epochs ):
-for batch_data in train_loader :
-batch_data = tf . cast ( batch_data , dtype = tf . float32 )
-d_loss , g_loss = self . train_step ( batch_data , optimizers )
-print (
-"Epoch: {} | disc_loss: {} | gen_loss: {} " . format (
-epoch , d_loss , g_loss
-))
-if epoch % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . discriminator . save_weights ( model_checkpoint_base_name . format ( 'discriminator' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
- g_lossfn ( real )
-
-
-
-
-
Calculates the Generator losses.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def g_lossfn ( self , real ):
- """Calculates the Generator losses.
- Args:
- real: real data.
- Returns:
- generator loss
- """
-# generating noise from a uniform distribution
-noise = tf . random . normal (( real . shape [ 0 ], self . noise_dim ), dtype = tf . float64 )
-fake = self . generator ( noise , training = True )
-logits_fake = self . discriminator ( fake , training = True )
-g_loss = - tf . reduce_mean ( logits_fake )
-return g_loss
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( train , batch_size )
-
-
-
-
-
Get real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def get_data_batch ( self , train , batch_size ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-buffer_size = len ( train )
-#tensor_data = pd.concat([x_train, y_train], axis=1)
-train_loader = tf . data . Dataset . from_tensor_slices ( train ) \
-. batch ( batch_size ) . shuffle ( buffer_size )
-return train_loader
-
-
-
-
-
-
-
-
-
-
-
- gradient_penalty ( real , fake )
-
-
-
-
-
Compute gradient penalty.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def gradient_penalty ( self , real , fake ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- Returns:
- gradient_penalty.
- """
-gp = gradient_penalty ( self . discriminator , real , fake , mode = Mode . DRAGAN )
-return gp
-
-
-
-
-
-
-
-
-
-
-
- train_step ( train_data , optimizers )
-
-
-
-
-
Perform a training step.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train_data
-
-
-
-
-
- required
-
-
-
- optimizers
-
-
-
-
generator and critic optimizers
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (critic_loss, generator_loss)
-
-
-
-
Critic and generator loss.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def train_step ( self , train_data , optimizers ):
- """Perform a training step.
- Args:
- train_data: training data
- optimizers: generator and critic optimizers
- Returns:
- (critic_loss, generator_loss): Critic and generator loss.
- """
-d_loss , g_loss = self . update_gradients ( train_data , * optimizers )
-return d_loss , g_loss
-
-
-
-
-
-
-
-
-
-
-
- update_gradients ( x , g_optimizer , d_optimizer )
-
-
-
-
-
Compute the gradients for Generator and Discriminator.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- x
- tensor
-
-
-
-
- required
-
-
-
- g_optimizer
- OptimizerV2
-
-
-
Optimizer for the generator model
-
-
- required
-
-
-
- c_optimizer
- OptimizerV2
-
-
-
Optimizer for the discriminator model
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/dragan/model.py
- def update_gradients ( self , x , g_optimizer , d_optimizer ):
- """Compute the gradients for Generator and Discriminator.
- Args:
- x (tf.tensor): real data event
- g_optimizer (tf.OptimizerV2): Optimizer for the generator model
- c_optimizer (tf.OptimizerV2): Optimizer for the discriminator model
- Returns:
- (discriminator loss, generator loss)
- """
-# Update the gradients of critic for n_critic times (Training the critic)
-for _ in range ( self . n_discriminator ):
-with tf . GradientTape () as d_tape :
-d_loss = self . d_lossfn ( x )
-# Get the gradients of the critic
-d_gradient = d_tape . gradient ( d_loss , self . discriminator . trainable_variables )
-# Update the weights of the critic using the optimizer
-d_optimizer . apply_gradients (
-zip ( d_gradient , self . discriminator . trainable_variables )
-)
-# Update the generator
-with tf . GradientTape () as g_tape :
-gen_loss = self . g_lossfn ( x )
-# Get the gradients of the generator
-gen_gradients = g_tape . gradient ( gen_loss , self . generator . trainable_variables )
-# Update the weights of the generator
-g_optimizer . apply_gradients (
-zip ( gen_gradients , self . generator . trainable_variables )
-)
-return d_loss , gen_loss
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
GAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- GAN
-
-
-
-
-
-
-
-
- Bases: BaseModel
-
-
-
Base class of GAN synthesizer models.
-The main methods are train (for fitting the synthesizer), save/load and sample (obtain synthetic records).
-Args:
- model_parameters (ModelParameters):
- Set of architectural parameters for model definition.
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- @typechecked
-class BaseGANModel ( BaseModel ):
- """
- Base class of GAN synthesizer models.
- The main methods are train (for fitting the synthesizer), save/load and sample (obtain synthetic records).
- Args:
- model_parameters (ModelParameters):
- Set of architectural parameters for model definition.
- """
-def __init__ (
-self ,
-model_parameters : ModelParameters
-):
-gpu_devices = tfconfig . list_physical_devices ( 'GPU' )
-if len ( gpu_devices ) > 0 :
-try :
-tfconfig . experimental . set_memory_growth ( gpu_devices [ 0 ], True )
-except ( ValueError , RuntimeError ):
-# Invalid device or cannot modify virtual devices once initialized.
-pass
-#Validate the provided model parameters
-if model_parameters . betas is not None :
-assert len ( model_parameters . betas ) == 2 , "Please provide the betas information as a tuple."
-self . batch_size = model_parameters . batch_size
-self . _set_lr ( model_parameters . lr )
-self . beta_1 = model_parameters . betas [ 0 ]
-self . beta_2 = model_parameters . betas [ 1 ]
-self . noise_dim = model_parameters . noise_dim
-self . data_dim = None
-self . layers_dim = model_parameters . layers_dim
-# Additional parameters for the CTGAN
-self . generator_dims = model_parameters . generator_dims
-self . critic_dims = model_parameters . critic_dims
-self . l2_scale = model_parameters . l2_scale
-self . latent_dim = model_parameters . latent_dim
-self . gp_lambda = model_parameters . gp_lambda
-self . pac = model_parameters . pac
-self . use_tanh = model_parameters . tanh
-self . processor = None
-if self . __MODEL__ in RegularModels . __members__ or \
-self . __MODEL__ == CTGANDataProcessor . SUPPORTED_MODEL :
-self . tau = model_parameters . tau_gs
-# pylint: disable=E1101
-def __call__ ( self , inputs , ** kwargs ):
-return self . model ( inputs = inputs , ** kwargs )
-# pylint: disable=C0103
-def _set_lr ( self , lr ):
-if isinstance ( lr , float ):
-self . g_lr = lr
-self . d_lr = lr
-elif isinstance ( lr ,( list , tuple )):
-assert len ( lr ) == 2 , "Please provide a two values array for the learning rates or a float."
-self . g_lr = lr [ 0 ]
-self . d_lr = lr [ 1 ]
-def define_gan ( self ):
- """Define the trainable model components.
- Optionally validate model structure with mock inputs and initialize optimizers."""
-raise NotImplementedError
-@property
-def model_parameters ( self ):
-"Returns the parameters of the model."
-return self . _model_parameters
-@property
-def model_name ( self ):
-"Returns the model (class) name."
-return self . __class__ . __name__
-def fit ( self ,
-data : Union [ DataFrame , array ],
-num_cols : Optional [ List [ str ]] = None ,
-cat_cols : Optional [ List [ str ]] = None ,
-train_arguments : Optional [ TrainParameters ] = None ) -> Union [ DataFrame , array ]:
- """
- Trains and fit a synthesizer model to a given input dataset.
- Args:
- data (Union[DataFrame, array]): Training data
- num_cols (Optional[List[str]]) : List with the names of the categorical columns
- cat_cols (Optional[List[str]]): List of names of categorical columns
- train_arguments (Optional[TrainParameters]): Training parameters
- Returns:
- Fitted synthesizer
- """
-if self . __MODEL__ in RegularModels . __members__ :
-self . processor = RegularDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ in TimeSeriesModels . __members__ :
-self . processor = TimeSeriesDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == CTGANDataProcessor . SUPPORTED_MODEL :
-n_clusters = train_arguments . n_clusters
-epsilon = train_arguments . epsilon
-self . processor = CTGANDataProcessor ( n_clusters = n_clusters , epsilon = epsilon ,
-num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == DoppelGANgerProcessor . SUPPORTED_MODEL :
-measurement_cols = train_arguments . measurement_cols
-sequence_length = train_arguments . sequence_length
-sample_length = train_arguments . sample_length
-self . processor = DoppelGANgerProcessor ( num_cols = num_cols , cat_cols = cat_cols ,
-measurement_cols = measurement_cols ,
-sequence_length = sequence_length ,
-sample_length = sample_length ,
-normalize_tanh = self . use_tanh ) . fit ( data )
-else :
-print ( f 'A DataProcessor is not available for the { self . __MODEL__ } .' )
-def sample ( self , n_samples : int ):
- """
- Generates samples from the trained synthesizer.
- Args:
- n_samples (int): Number of rows to generated.
- Returns:
- synth_sample (pandas.DataFrame): generated synthetic samples.
- """
-steps = n_samples // self . batch_size + 1
-data = []
-for _ in tqdm . trange ( steps , desc = 'Synthetic data generation' ):
-z = random . uniform ([ self . batch_size , self . noise_dim ], dtype = tf . dtypes . float32 )
-records = self . generator ( z , training = False ) . numpy ()
-data . append ( records )
-return self . processor . inverse_transform ( array ( vstack ( data )))
-def save ( self , path ):
- """
- Saves a synthesizer as a pickle.
- Args:
- path (str): Path to write the synthesizer as a pickle object.
- """
-#Save only the generator?
-if self . __MODEL__ == 'WGAN' or self . __MODEL__ == 'WGAN_GP' or self . __MODEL__ == 'CWGAN_GP' :
-del self . critic
-make_keras_picklable ()
-dump ( self , path )
-@classmethod
-def load ( cls , path ):
- """
- Loads a saved synthesizer from a pickle.
- Args:
- path (str): Path to read the synthesizer pickle from.
- """
-gpu_devices = tfconfig . list_physical_devices ( 'GPU' )
-if len ( gpu_devices ) > 0 :
-try :
-tfconfig . experimental . set_memory_growth ( gpu_devices [ 0 ], True )
-except ( ValueError , RuntimeError ):
-# Invalid device or cannot modify virtual devices once initialized.
-pass
-synth = load ( path )
-return synth
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- model_name
- property
-
-
-
-
-
-
-
Returns the model (class) name.
-
-
-
-
-
-
-
-
-
- model_parameters
- property
-
-
-
-
-
-
-
Returns the parameters of the model.
-
-
-
-
-
-
-
-
-
-
-
-
- define_gan ()
-
-
-
-
-
Define the trainable model components.
-
Optionally validate model structure with mock inputs and initialize optimizers.
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def define_gan ( self ):
- """Define the trainable model components.
- Optionally validate model structure with mock inputs and initialize optimizers."""
-raise NotImplementedError
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , num_cols = None , cat_cols = None , train_arguments = None )
-
-
-
-
-
Trains and fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- Union [DataFrame , array ]
-
-
-
-
- required
-
-
-
- num_cols
- Optional[List[str]])
-
-
-
-
List with the names of the categorical columns
-
-
- None
-
-
-
- cat_cols
- Optional [List [str]]
-
-
-
List of names of categorical columns
-
-
- None
-
-
-
- train_arguments
- Optional [TrainParameters ]
-
-
-
-
- None
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- Union [DataFrame , array ]
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def fit ( self ,
-data : Union [ DataFrame , array ],
-num_cols : Optional [ List [ str ]] = None ,
-cat_cols : Optional [ List [ str ]] = None ,
-train_arguments : Optional [ TrainParameters ] = None ) -> Union [ DataFrame , array ]:
- """
- Trains and fit a synthesizer model to a given input dataset.
- Args:
- data (Union[DataFrame, array]): Training data
- num_cols (Optional[List[str]]) : List with the names of the categorical columns
- cat_cols (Optional[List[str]]): List of names of categorical columns
- train_arguments (Optional[TrainParameters]): Training parameters
- Returns:
- Fitted synthesizer
- """
-if self . __MODEL__ in RegularModels . __members__ :
-self . processor = RegularDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ in TimeSeriesModels . __members__ :
-self . processor = TimeSeriesDataProcessor ( num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == CTGANDataProcessor . SUPPORTED_MODEL :
-n_clusters = train_arguments . n_clusters
-epsilon = train_arguments . epsilon
-self . processor = CTGANDataProcessor ( n_clusters = n_clusters , epsilon = epsilon ,
-num_cols = num_cols , cat_cols = cat_cols ) . fit ( data )
-elif self . __MODEL__ == DoppelGANgerProcessor . SUPPORTED_MODEL :
-measurement_cols = train_arguments . measurement_cols
-sequence_length = train_arguments . sequence_length
-sample_length = train_arguments . sample_length
-self . processor = DoppelGANgerProcessor ( num_cols = num_cols , cat_cols = cat_cols ,
-measurement_cols = measurement_cols ,
-sequence_length = sequence_length ,
-sample_length = sample_length ,
-normalize_tanh = self . use_tanh ) . fit ( data )
-else :
-print ( f 'A DataProcessor is not available for the { self . __MODEL__ } .' )
-
-
-
-
-
-
-
-
-
-
-
- load ( path )
- classmethod
-
-
-
-
-
-
-
Loads a saved synthesizer from a pickle.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- path
- str
-
-
-
-
Path to read the synthesizer pickle from.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- @classmethod
-def load ( cls , path ):
- """
- Loads a saved synthesizer from a pickle.
- Args:
- path (str): Path to read the synthesizer pickle from.
- """
-gpu_devices = tfconfig . list_physical_devices ( 'GPU' )
-if len ( gpu_devices ) > 0 :
-try :
-tfconfig . experimental . set_memory_growth ( gpu_devices [ 0 ], True )
-except ( ValueError , RuntimeError ):
-# Invalid device or cannot modify virtual devices once initialized.
-pass
-synth = load ( path )
-return synth
-
-
-
-
-
-
-
-
-
-
-
- sample ( n_samples )
-
-
-
-
-
Generates samples from the trained synthesizer.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- n_samples
- int
-
-
-
-
Number of rows to generated.
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
-Name Type
- Description
-
-
-
-
-synth_sample
- DataFrame
-
-
-
generated synthetic samples.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def sample ( self , n_samples : int ):
- """
- Generates samples from the trained synthesizer.
- Args:
- n_samples (int): Number of rows to generated.
- Returns:
- synth_sample (pandas.DataFrame): generated synthetic samples.
- """
-steps = n_samples // self . batch_size + 1
-data = []
-for _ in tqdm . trange ( steps , desc = 'Synthetic data generation' ):
-z = random . uniform ([ self . batch_size , self . noise_dim ], dtype = tf . dtypes . float32 )
-records = self . generator ( z , training = False ) . numpy ()
-data . append ( records )
-return self . processor . inverse_transform ( array ( vstack ( data )))
-
-
-
-
-
-
-
-
-
-
-
- save ( path )
-
-
-
-
-
Saves a synthesizer as a pickle.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- path
- str
-
-
-
-
Path to write the synthesizer as a pickle object.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/base.py
- def save ( self , path ):
- """
- Saves a synthesizer as a pickle.
- Args:
- path (str): Path to write the synthesizer as a pickle object.
- """
-#Save only the generator?
-if self . __MODEL__ == 'WGAN' or self . __MODEL__ == 'WGAN_GP' or self . __MODEL__ == 'CWGAN_GP' :
-del self . critic
-make_keras_picklable ()
-dump ( self , path )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Vanilllagan - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Vanilllagan
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/vanillagan/model.py
- class VanilllaGAN ( BaseGANModel ):
-__MODEL__ = 'GAN'
-def __init__ ( self , model_parameters ):
-super () . __init__ ( model_parameters )
-def define_gan ( self , activation_info : Optional [ NamedTuple ]):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,)
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the discriminator
-self . discriminator . compile ( loss = 'binary_crossentropy' ,
-optimizer = d_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-z = Input ( shape = ( self . noise_dim ,))
-record = self . generator ( z )
-# For the combined model we will only train the generator
-self . discriminator . trainable = False
-# The discriminator takes generated images as input and determines validity
-validity = self . discriminator ( record )
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-self . _model = Model ( z , validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = g_optimizer )
-def get_data_batch ( self , train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data
- batch_size: batch size
- seed (int, optional):Defaults to 0.
- Returns:
- data batch
- """
-# # random sampling - some samples will have excessively low or high sampling, but easy to implement
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ], cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized
- train_arguments: GAN training arguments.
- num_cols (List[str]): List of columns of the data object to be handled as numerical
- cat_cols (List[str]): List of columns of the data object to be handled as categorical
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . define_gan ( self . processor . col_transform_info )
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = np . zeros (( self . batch_size , 1 ))
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_data = self . get_data_batch ( processed_data , self . batch_size )
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Generate a batch of events
-gen_data = self . generator ( noise , training = True )
-# Train the discriminator
-d_loss_real = self . discriminator . train_on_batch ( batch_data , valid )
-d_loss_fake = self . discriminator . train_on_batch ( gen_data , fake )
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Train the generator (to have the discriminator label samples as valid)
-g_loss = self . _model . train_on_batch ( noise , valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-# If at save interval => save generated events
-if epoch % train_arguments . sample_interval == 0 :
-#Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . discriminator . save_weights ( model_checkpoint_base_name . format ( 'discriminator' , epoch ))
-#Here is generating the data
-z = tf . random . normal (( 432 , self . noise_dim ))
-gen_data = self . generator ( z )
-print ( 'generated_data' )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (generator_optimizer, critic_optimizer)
-
-
-
-
Generator and critic optimizers
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/vanillagan/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ]):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,)
-self . discriminator = Discriminator ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-d_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the discriminator
-self . discriminator . compile ( loss = 'binary_crossentropy' ,
-optimizer = d_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-z = Input ( shape = ( self . noise_dim ,))
-record = self . generator ( z )
-# For the combined model we will only train the generator
-self . discriminator . trainable = False
-# The discriminator takes generated images as input and determines validity
-validity = self . discriminator ( record )
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-self . _model = Model ( z , validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = g_optimizer )
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
A pandas DataFrame or a Numpy array with the data to be synthesized
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/vanillagan/model.py
- def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ], cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized
- train_arguments: GAN training arguments.
- num_cols (List[str]): List of columns of the data object to be handled as numerical
- cat_cols (List[str]): List of columns of the data object to be handled as categorical
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . define_gan ( self . processor . col_transform_info )
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = np . zeros (( self . batch_size , 1 ))
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-# ---------------------
-# Train Discriminator
-# ---------------------
-batch_data = self . get_data_batch ( processed_data , self . batch_size )
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Generate a batch of events
-gen_data = self . generator ( noise , training = True )
-# Train the discriminator
-d_loss_real = self . discriminator . train_on_batch ( batch_data , valid )
-d_loss_fake = self . discriminator . train_on_batch ( gen_data , fake )
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Train the generator (to have the discriminator label samples as valid)
-g_loss = self . _model . train_on_batch ( noise , valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-# If at save interval => save generated events
-if epoch % train_arguments . sample_interval == 0 :
-#Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . discriminator . save_weights ( model_checkpoint_base_name . format ( 'discriminator' , epoch ))
-#Here is generating the data
-z = tf . random . normal (( 432 , self . noise_dim ))
-gen_data = self . generator ( z )
-print ( 'generated_data' )
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( train , batch_size , seed = 0 )
-
-
-
-
-
Get real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/vanillagan/model.py
- def get_data_batch ( self , train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data
- batch_size: batch size
- seed (int, optional):Defaults to 0.
- Returns:
- data batch
- """
-# # random sampling - some samples will have excessively low or high sampling, but easy to implement
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
WGAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- WGAN
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgan/model.py
- class WGAN ( BaseGANModel ):
-__MODEL__ = 'WGAN'
-def __init__ ( self , model_parameters , n_critic , clip_value = 0.01 ):
-# As recommended in WGAN paper - https://arxiv.org/abs/1701.07875
-# WGAN-GP - WGAN with Gradient Penalty
-self . n_critic = n_critic
-self . clip_value = clip_value
-super () . __init__ ( model_parameters )
-def wasserstein_loss ( self , y_true , y_pred ):
- """Calculate wasserstein loss.
- Args:
- y_true: ground truth.
- y_pred: predictions.
- Returns:
- wasserstein loss.
- """
-return K . mean ( y_true * y_pred )
-def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers.
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-critic_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the critic
-self . critic . compile ( loss = self . wasserstein_loss ,
-optimizer = critic_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-z = Input ( shape = ( self . noise_dim ,))
-record = self . generator ( z )
-# The discriminator takes generated images as input and determines validity
-validity = self . critic ( record )
-# For the combined model we will only train the generator
-self . critic . trainable = False
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-#For the WGAN model use the Wassertein loss
-self . _model = Model ( z , validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = optimizer )
-def get_data_batch ( self , train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ],
-cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized.
- train_arguments: GAN training arguments.
- num_cols (List[str]): List of columns of the data object to be handled as numerical.
- cat_cols (List[str]): List of columns of the data object to be handled as categorical.
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . define_gan ( self . processor . col_transform_info )
-#Create a summary file
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '.' , 'summaries' , 'train' ))
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = - np . ones (( self . batch_size , 1 ))
-with train_summary_writer . as_default ():
-for epoch in trange ( train_arguments . epochs , desc = 'Epoch Iterations' ):
-for _ in range ( iterations ):
-for _ in range ( self . n_critic ):
-# ---------------------
-# Train the Critic
-# ---------------------
-batch_data = self . get_data_batch ( processed_data , self . batch_size )
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Generate a batch of events
-gen_data = self . generator ( noise )
-# Train the Critic
-d_loss_real = self . critic . train_on_batch ( batch_data , valid )
-d_loss_fake = self . critic . train_on_batch ( gen_data , fake )
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-for l in self . critic . layers :
-weights = l . get_weights ()
-weights = [ np . clip ( w , - self . clip_value , self . clip_value ) for w in weights ]
-l . set_weights ( weights )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Train the generator (to have the critic label samples as valid)
-g_loss = self . _model . train_on_batch ( noise , valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-#If at save interval => save generated events
-if epoch % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (generator_optimizer, critic_optimizer)
-
-
-
-
Generator and critic optimizers.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgan/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers.
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-critic_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-# Build and compile the critic
-self . critic . compile ( loss = self . wasserstein_loss ,
-optimizer = critic_optimizer ,
-metrics = [ 'accuracy' ])
-# The generator takes noise as input and generates imgs
-z = Input ( shape = ( self . noise_dim ,))
-record = self . generator ( z )
-# The discriminator takes generated images as input and determines validity
-validity = self . critic ( record )
-# For the combined model we will only train the generator
-self . critic . trainable = False
-# The combined model (stacked generator and discriminator)
-# Trains the generator to fool the discriminator
-#For the WGAN model use the Wassertein loss
-self . _model = Model ( z , validity )
-self . _model . compile ( loss = 'binary_crossentropy' , optimizer = optimizer )
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
A pandas DataFrame or a Numpy array with the data to be synthesized.
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical.
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgan/model.py
- def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ],
-cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized.
- train_arguments: GAN training arguments.
- num_cols (List[str]): List of columns of the data object to be handled as numerical.
- cat_cols (List[str]): List of columns of the data object to be handled as categorical.
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-self . define_gan ( self . processor . col_transform_info )
-#Create a summary file
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '.' , 'summaries' , 'train' ))
-# Adversarial ground truths
-valid = np . ones (( self . batch_size , 1 ))
-fake = - np . ones (( self . batch_size , 1 ))
-with train_summary_writer . as_default ():
-for epoch in trange ( train_arguments . epochs , desc = 'Epoch Iterations' ):
-for _ in range ( iterations ):
-for _ in range ( self . n_critic ):
-# ---------------------
-# Train the Critic
-# ---------------------
-batch_data = self . get_data_batch ( processed_data , self . batch_size )
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Generate a batch of events
-gen_data = self . generator ( noise )
-# Train the Critic
-d_loss_real = self . critic . train_on_batch ( batch_data , valid )
-d_loss_fake = self . critic . train_on_batch ( gen_data , fake )
-d_loss = 0.5 * np . add ( d_loss_real , d_loss_fake )
-for l in self . critic . layers :
-weights = l . get_weights ()
-weights = [ np . clip ( w , - self . clip_value , self . clip_value ) for w in weights ]
-l . set_weights ( weights )
-# ---------------------
-# Train Generator
-# ---------------------
-noise = tf . random . normal (( self . batch_size , self . noise_dim ))
-# Train the generator (to have the critic label samples as valid)
-g_loss = self . _model . train_on_batch ( noise , valid )
-# Plot the progress
-print ( " %d [D loss: %f , acc.: %.2f%% ] [G loss: %f ]" % ( epoch , d_loss [ 0 ], 100 * d_loss [ 1 ], g_loss ))
-#If at save interval => save generated events
-if epoch % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( train , batch_size , seed = 0 )
-
-
-
-
-
Get real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgan/model.py
- def get_data_batch ( self , train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-
-
-
-
-
-
-
-
-
-
-
- wasserstein_loss ( y_true , y_pred )
-
-
-
-
-
Calculate wasserstein loss.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- y_true
-
-
-
-
-
- required
-
-
-
- y_pred
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgan/model.py
- def wasserstein_loss ( self , y_true , y_pred ):
- """Calculate wasserstein loss.
- Args:
- y_true: ground truth.
- y_pred: predictions.
- Returns:
- wasserstein loss.
- """
-return K . mean ( y_true * y_pred )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
WGAN_GP - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- WGAN_GP
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- class WGAN_GP ( BaseGANModel ):
-__MODEL__ = 'WGAN_GP'
-def __init__ ( self , model_parameters , n_generator : int = 1 , n_critic : int = 1 , gradient_penalty_weight : int = 10 ):
-# As recommended in WGAN paper - https://arxiv.org/abs/1701.07875
-# WGAN-GP - WGAN with Gradient Penalty
-self . n_critic = n_critic
-self . n_generator = n_generator
-self . gradient_penalty_weight = gradient_penalty_weight
-super () . __init__ ( model_parameters )
-def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers.
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-c_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-return g_optimizer , c_optimizer
-def gradient_penalty ( self , real , fake ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- Returns:
- gradient_penalty.
- """
-epsilon = tf . random . uniform ([ real . shape [ 0 ], 1 ], minval = 0.0 , maxval = 1.0 , dtype = tf . dtypes . float32 )
-x_hat = epsilon * real + ( 1 - epsilon ) * fake
-with tf . GradientTape () as t :
-t . watch ( x_hat )
-d_hat = self . critic ( x_hat )
-gradients = t . gradient ( d_hat , x_hat )
-ddx = tf . sqrt ( tf . reduce_sum ( gradients ** 2 ))
-d_regularizer = tf . reduce_mean (( ddx - 1.0 ) ** 2 )
-return d_regularizer
-@tf . function
-def update_gradients ( self , x , g_optimizer , c_optimizer ):
- """Compute and apply the gradients for both the Generator and the Critic.
- Args:
- x: real data event
- g_optimizer: generator optimizer
- c_optimizer: critic optimizer
- Returns:
- (critic loss, generator loss)
- """
-for _ in range ( self . n_critic ):
-with tf . GradientTape () as d_tape :
-critic_loss = self . c_lossfn ( x )
-# Get the gradients of the critic
-d_gradient = d_tape . gradient ( critic_loss , self . critic . trainable_variables )
-# Update the weights of the critic using the optimizer
-c_optimizer . apply_gradients (
-zip ( d_gradient , self . critic . trainable_variables )
-)
-##Add here the n_generator
-# Update the generator
-for _ in range ( self . n_generator ):
-with tf . GradientTape () as g_tape :
-gen_loss = self . g_lossfn ( x )
-# Get the gradients of the generator
-gen_gradients = g_tape . gradient ( gen_loss , self . generator . trainable_variables )
-# Update the weights of the generator
-g_optimizer . apply_gradients (
-zip ( gen_gradients , self . generator . trainable_variables )
-)
-return critic_loss , gen_loss
-def c_lossfn ( self , real ):
- """Compute critic loss.
- Args:
- real: real data
- Returns:
- critic loss
- """
-# generating noise from a uniform distribution
-noise = tf . random . normal ([ real . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-# run noise through generator
-fake = self . generator ( noise )
-# discriminate x and x_gen
-logits_real = self . critic ( real )
-logits_fake = self . critic ( fake )
-# gradient penalty
-gp = self . gradient_penalty ( real , fake )
-# getting the loss of the critic.
-c_loss = ( tf . reduce_mean ( logits_fake )
-- tf . reduce_mean ( logits_real )
-+ gp * self . gradient_penalty_weight )
-return c_loss
-def g_lossfn ( self , real ):
- """Compute generator loss.
- Args:
- real: A real sample
- fake: A fake sample
- fak2: A second fake sample
- Returns:
- Loss of the generator
- """
-# generating noise from a uniform distribution
-noise = tf . random . normal ([ real . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-fake = self . generator ( noise )
-logits_fake = self . critic ( fake )
-g_loss = - tf . reduce_mean ( logits_fake )
-return g_loss
-def get_data_batch ( self , train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-def train_step ( self , train_data , optimizers ):
- """Perform a training step.
- Args:
- train_data: training data
- optimizers: generator and critic optimizers
- Returns:
- (critic_loss, generator_loss): Critic and generator loss.
- """
-cri_loss , ge_loss = self . update_gradients ( train_data , * optimizers )
-return cri_loss , ge_loss
-def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ], cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized.
- train_arguments: GAN training arguments.
- num_cols (List[str]): List of columns of the data object to be handled as numerical.
- cat_cols (List[str]): List of columns of the data object to be handled as categorical.
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-optimizers = self . define_gan ( self . processor . col_transform_info )
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-# Create a summary file
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '..\wgan_gp_test' , 'summaries' , 'train' ))
-with train_summary_writer . as_default ():
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-batch_data = self . get_data_batch ( processed_data , self . batch_size ) . astype ( np . float32 )
-cri_loss , ge_loss = self . train_step ( batch_data , optimizers )
-print (
-"Epoch: {} | disc_loss: {} | gen_loss: {} " . format (
-epoch , cri_loss , ge_loss
-))
-if epoch % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- c_lossfn ( real )
-
-
-
-
-
Compute critic loss.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def c_lossfn ( self , real ):
- """Compute critic loss.
- Args:
- real: real data
- Returns:
- critic loss
- """
-# generating noise from a uniform distribution
-noise = tf . random . normal ([ real . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-# run noise through generator
-fake = self . generator ( noise )
-# discriminate x and x_gen
-logits_real = self . critic ( real )
-logits_fake = self . critic ( fake )
-# gradient penalty
-gp = self . gradient_penalty ( real , fake )
-# getting the loss of the critic.
-c_loss = ( tf . reduce_mean ( logits_fake )
-- tf . reduce_mean ( logits_real )
-+ gp * self . gradient_penalty_weight )
-return c_loss
-
-
-
-
-
-
-
-
-
-
-
- define_gan ( activation_info = None )
-
-
-
-
-
Define the trainable model components.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- activation_info
- Optional [NamedTuple ]
-
-
-
-
- None
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (generator_optimizer, critic_optimizer)
-
-
-
-
Generator and critic optimizers.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def define_gan ( self , activation_info : Optional [ NamedTuple ] = None ):
- """Define the trainable model components.
- Args:
- activation_info (Optional[NamedTuple], optional): Defaults to None.
- Returns:
- (generator_optimizer, critic_optimizer): Generator and critic optimizers.
- """
-self . generator = Generator ( self . batch_size ) . \
-build_model ( input_shape = ( self . noise_dim ,), dim = self . layers_dim , data_dim = self . data_dim ,
-activation_info = activation_info , tau = self . tau )
-self . critic = Critic ( self . batch_size ) . \
-build_model ( input_shape = ( self . data_dim ,), dim = self . layers_dim )
-g_optimizer = Adam ( self . g_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-c_optimizer = Adam ( self . d_lr , beta_1 = self . beta_1 , beta_2 = self . beta_2 )
-return g_optimizer , c_optimizer
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols , cat_cols )
-
-
-
-
-
Fit a synthesizer model to a given input dataset.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
-
-
-
-
A pandas DataFrame or a Numpy array with the data to be synthesized.
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
GAN training arguments.
-
-
- required
-
-
-
- num_cols
- List [str]
-
-
-
List of columns of the data object to be handled as numerical.
-
-
- required
-
-
-
- cat_cols
- List [str]
-
-
-
List of columns of the data object to be handled as categorical.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def fit ( self , data , train_arguments : TrainParameters , num_cols : List [ str ], cat_cols : List [ str ]):
- """Fit a synthesizer model to a given input dataset.
- Args:
- data: A pandas DataFrame or a Numpy array with the data to be synthesized.
- train_arguments: GAN training arguments.
- num_cols (List[str]): List of columns of the data object to be handled as numerical.
- cat_cols (List[str]): List of columns of the data object to be handled as categorical.
- """
-super () . fit ( data , num_cols , cat_cols )
-processed_data = self . processor . transform ( data )
-self . data_dim = processed_data . shape [ 1 ]
-optimizers = self . define_gan ( self . processor . col_transform_info )
-iterations = int ( abs ( data . shape [ 0 ] / self . batch_size ) + 1 )
-# Create a summary file
-train_summary_writer = tf . summary . create_file_writer ( path . join ( '..\wgan_gp_test' , 'summaries' , 'train' ))
-with train_summary_writer . as_default ():
-for epoch in trange ( train_arguments . epochs ):
-for _ in range ( iterations ):
-batch_data = self . get_data_batch ( processed_data , self . batch_size ) . astype ( np . float32 )
-cri_loss , ge_loss = self . train_step ( batch_data , optimizers )
-print (
-"Epoch: {} | disc_loss: {} | gen_loss: {} " . format (
-epoch , cri_loss , ge_loss
-))
-if epoch % train_arguments . sample_interval == 0 :
-# Test here data generation step
-# save model checkpoints
-if path . exists ( './cache' ) is False :
-os . mkdir ( './cache' )
-model_checkpoint_base_name = './cache/' + train_arguments . cache_prefix + '_ {} _model_weights_step_ {} .h5'
-self . generator . save_weights ( model_checkpoint_base_name . format ( 'generator' , epoch ))
-self . critic . save_weights ( model_checkpoint_base_name . format ( 'critic' , epoch ))
-
-
-
-
-
-
-
-
-
-
-
- g_lossfn ( real )
-
-
-
-
-
Compute generator loss.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
- fak2
-
-
-
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
Loss of the generator
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def g_lossfn ( self , real ):
- """Compute generator loss.
- Args:
- real: A real sample
- fake: A fake sample
- fak2: A second fake sample
- Returns:
- Loss of the generator
- """
-# generating noise from a uniform distribution
-noise = tf . random . normal ([ real . shape [ 0 ], self . noise_dim ], dtype = tf . dtypes . float32 )
-fake = self . generator ( noise )
-logits_fake = self . critic ( fake )
-g_loss = - tf . reduce_mean ( logits_fake )
-return g_loss
-
-
-
-
-
-
-
-
-
-
-
- get_data_batch ( train , batch_size , seed = 0 )
-
-
-
-
-
Get real data batches from the passed data object.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train
-
-
-
-
-
- required
-
-
-
- batch_size
-
-
-
-
-
- required
-
-
-
- seed
- int
-
-
-
-
-
- 0
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def get_data_batch ( self , train , batch_size , seed = 0 ):
- """Get real data batches from the passed data object.
- Args:
- train: real data.
- batch_size: batch size.
- seed (int, optional):Defaults to 0.
- Returns:
- data batch.
- """
-# np.random.seed(seed)
-# x = train.loc[ np.random.choice(train.index, batch_size) ].values
-# iterate through shuffled indices, so every sample gets covered evenly
-start_i = ( batch_size * seed ) % len ( train )
-stop_i = start_i + batch_size
-shuffle_seed = ( batch_size * seed ) // len ( train )
-np . random . seed ( shuffle_seed )
-train_ix = np . random . choice ( train . shape [ 0 ], replace = False , size = len ( train )) # wasteful to shuffle every time
-train_ix = list ( train_ix ) + list ( train_ix ) # duplicate to cover ranges past the end of the set
-return train [ train_ix [ start_i : stop_i ]]
-
-
-
-
-
-
-
-
-
-
-
- gradient_penalty ( real , fake )
-
-
-
-
-
Compute gradient penalty.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- real
-
-
-
-
-
- required
-
-
-
- fake
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def gradient_penalty ( self , real , fake ):
- """Compute gradient penalty.
- Args:
- real: real event.
- fake: fake event.
- Returns:
- gradient_penalty.
- """
-epsilon = tf . random . uniform ([ real . shape [ 0 ], 1 ], minval = 0.0 , maxval = 1.0 , dtype = tf . dtypes . float32 )
-x_hat = epsilon * real + ( 1 - epsilon ) * fake
-with tf . GradientTape () as t :
-t . watch ( x_hat )
-d_hat = self . critic ( x_hat )
-gradients = t . gradient ( d_hat , x_hat )
-ddx = tf . sqrt ( tf . reduce_sum ( gradients ** 2 ))
-d_regularizer = tf . reduce_mean (( ddx - 1.0 ) ** 2 )
-return d_regularizer
-
-
-
-
-
-
-
-
-
-
-
- train_step ( train_data , optimizers )
-
-
-
-
-
Perform a training step.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- train_data
-
-
-
-
-
- required
-
-
-
- optimizers
-
-
-
-
generator and critic optimizers
-
-
- required
-
-
-
-
-
-
-
-
Returns:
-
-
-
- Type
- Description
-
-
-
-
-
- (critic_loss, generator_loss)
-
-
-
-
Critic and generator loss.
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- def train_step ( self , train_data , optimizers ):
- """Perform a training step.
- Args:
- train_data: training data
- optimizers: generator and critic optimizers
- Returns:
- (critic_loss, generator_loss): Critic and generator loss.
- """
-cri_loss , ge_loss = self . update_gradients ( train_data , * optimizers )
-return cri_loss , ge_loss
-
-
-
-
-
-
-
-
-
-
-
- update_gradients ( x , g_optimizer , c_optimizer )
-
-
-
-
-
Compute and apply the gradients for both the Generator and the Critic.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- x
-
-
-
-
-
- required
-
-
-
- g_optimizer
-
-
-
-
-
- required
-
-
-
- c_optimizer
-
-
-
-
-
- required
-
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/regular/wgangp/model.py
- @tf . function
-def update_gradients ( self , x , g_optimizer , c_optimizer ):
- """Compute and apply the gradients for both the Generator and the Critic.
- Args:
- x: real data event
- g_optimizer: generator optimizer
- c_optimizer: critic optimizer
- Returns:
- (critic loss, generator loss)
- """
-for _ in range ( self . n_critic ):
-with tf . GradientTape () as d_tape :
-critic_loss = self . c_lossfn ( x )
-# Get the gradients of the critic
-d_gradient = d_tape . gradient ( critic_loss , self . critic . trainable_variables )
-# Update the weights of the critic using the optimizer
-c_optimizer . apply_gradients (
-zip ( d_gradient , self . critic . trainable_variables )
-)
-##Add here the n_generator
-# Update the generator
-for _ in range ( self . n_generator ):
-with tf . GradientTape () as g_tape :
-gen_loss = self . g_lossfn ( x )
-# Get the gradients of the generator
-gen_gradients = g_tape . gradient ( gen_loss , self . generator . trainable_variables )
-# Update the weights of the generator
-g_optimizer . apply_gradients (
-zip ( gen_gradients , self . generator . trainable_variables )
-)
-return critic_loss , gen_loss
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
DoppelGANger - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- DoppelGANger
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
DoppelGANger model.
-Based on the paper https://dl.acm.org/doi/pdf/10.1145/3419394.3423643.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- model_parameters
- ModelParameters
-
-
-
Parameters used to create the DoppelGANger model.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/doppelganger/model.py
- class DoppelGANger ( BaseGANModel ):
- """
- DoppelGANger model.
- Based on the paper https://dl.acm.org/doi/pdf/10.1145/3419394.3423643.
- Args:
- model_parameters: Parameters used to create the DoppelGANger model.
- """
-__MODEL__ = 'DoppelGANger'
-def __init__ ( self , model_parameters : ModelParameters ):
-super () . __init__ ( model_parameters )
-self . _model_parameters = model_parameters
-self . _gan_model = None
-self . _tf_session = None
-self . _sequence_length = None
-tf . compat . v1 . disable_eager_execution ()
-def fit ( self , data : DataFrame ,
-train_arguments : TrainParameters ,
-num_cols : list [ str ] | None = None ,
-cat_cols : list [ str ] | None = None ):
- """
- Fits the DoppelGANger model.
- Args:
- data: A pandas DataFrame with the data to be synthesized.
- train_arguments: DoppelGANger training arguments.
- num_cols: List of columns to be handled as numerical
- cat_cols: List of columns to be handled as categorical
- """
-super () . fit ( data = data , num_cols = num_cols , cat_cols = cat_cols , train_arguments = train_arguments )
-self . _sequence_length = train_arguments . sequence_length
-self . _sample_length = train_arguments . sample_length
-self . _rounds = train_arguments . rounds
-if data . shape [ 0 ] % self . _sequence_length != 0 :
-raise ValueError ( "The number of samples must be a multiple of the sequence length." )
-if self . _sequence_length % self . _sample_length != 0 :
-raise ValueError ( "The sequence length must be a multiple of the sample length." )
-data_features , data_attributes = self . processor . transform ( data )
-measurement_cols_metadata = self . processor . measurement_cols_metadata
-attribute_cols_metadata = self . processor . attribute_cols_metadata
-generator = DoppelGANgerGenerator (
-feed_back = False ,
-noise = True ,
-use_tanh = self . use_tanh ,
-measurement_cols_metadata = measurement_cols_metadata ,
-attribute_cols_metadata = attribute_cols_metadata ,
-sample_len = self . _sample_length )
-discriminator = Discriminator ()
-attr_discriminator = AttrDiscriminator ()
-self . _tf_session = tf . compat . v1 . Session ()
-with self . _tf_session . as_default () as sess :
-self . _gan_model = DoppelGANgerNetwork (
-sess = sess ,
-epoch = train_arguments . epochs ,
-batch_size = self . batch_size ,
-data_feature = data_features ,
-data_attribute = data_attributes ,
-attribute_cols_metadata = attribute_cols_metadata ,
-sample_len = self . _sample_length ,
-generator = generator ,
-discriminator = discriminator ,
-rounds = self . _rounds ,
-attr_discriminator = attr_discriminator ,
-d_gp_coe = self . gp_lambda ,
-attr_d_gp_coe = self . gp_lambda ,
-g_attr_d_coe = self . gp_lambda ,
-num_packing = self . pac ,
-attribute_latent_dim = self . latent_dim ,
-feature_latent_dim = self . latent_dim ,
-fix_feature_network = False ,
-g_lr = self . g_lr ,
-g_beta1 = self . beta_1 ,
-d_lr = self . d_lr ,
-d_beta1 = self . beta_1 ,
-attr_d_lr = self . d_lr ,
-attr_d_beta1 = self . beta_1 )
-self . _gan_model . build ()
-self . _gan_model . train ()
-def sample ( self , n_samples : int ):
- """
- Samples new data from the DoppelGANger.
- Args:
- n_samples: Number of samples to be generated.
- """
-if n_samples <= 0 :
-raise ValueError ( "Invalid number of samples." )
-real_attribute_input_noise = self . _gan_model . gen_attribute_input_noise ( n_samples )
-addi_attribute_input_noise = self . _gan_model . gen_attribute_input_noise ( n_samples )
-length = int ( self . _sequence_length / self . _sample_length )
-feature_input_noise = self . _gan_model . gen_feature_input_noise ( n_samples , length = length )
-input_data = self . _gan_model . gen_feature_input_data_free ( n_samples )
-with self . _tf_session . as_default () as sess :
-self . _gan_model . sess = sess
-data_features , data_attributes , gen_flags , _ = self . _gan_model . sample_from (
-real_attribute_input_noise , addi_attribute_input_noise ,
-feature_input_noise , input_data )
-return self . processor . inverse_transform ( data_features , data_attributes , gen_flags )
-def save ( self , path ):
- """
- Save the DoppelGANger model in a directory.
- Args:
- path: Path of the directory where the files will be saved.
- """
-saver = tf . compat . v1 . train . Saver ()
-with self . _tf_session . as_default () as sess :
-saver . save ( sess , os . path . join ( path , "doppelganger" ), write_meta_graph = False )
-self . _gan_model . save ( os . path . join ( path , "doppelganger_network.pkl" ))
-dump ({
-"processor" : self . processor . __dict__ ,
-"measurement_cols_metadata" : self . processor . measurement_cols_metadata ,
-"attribute_cols_metadata" : self . processor . attribute_cols_metadata ,
-"_sequence_length" : self . _sequence_length ,
-"_sample_length" : self . _sample_length
-}, os . path . join ( path , "doppelganger_metadata.pkl" ))
-@staticmethod
-def load ( path ):
- """
- Load the DoppelGANger model from a directory.
- Only the required components to sample new data are loaded.
- Args:
- class_dict: Path of the directory where the files were saved.
- """
-dp_model = DoppelGANger ( ModelParameters ())
-dp_network_parms = load ( os . path . join ( path , "doppelganger_network.pkl" ))
-dp_metadata = load ( os . path . join ( path , "doppelganger_metadata.pkl" ))
-dp_model . processor = DoppelGANgerProcessor ()
-dp_model . processor . __dict__ = dp_metadata [ "processor" ]
-dp_model . _sequence_length = dp_metadata [ "_sequence_length" ]
-dp_model . _sample_length = dp_metadata [ "_sample_length" ]
-generator = DoppelGANgerGenerator (
-feed_back = False ,
-noise = True ,
-measurement_cols_metadata = dp_metadata [ "measurement_cols_metadata" ],
-attribute_cols_metadata = dp_metadata [ "attribute_cols_metadata" ],
-sample_len = dp_network_parms [ "sample_len" ])
-discriminator = Discriminator ()
-attr_discriminator = AttrDiscriminator ()
-with tf . compat . v1 . Session () . as_default () as sess :
-dp_model . _gan_model = DoppelGANgerNetwork (
-sess = sess ,
-epoch = dp_network_parms [ "epoch" ],
-batch_size = dp_network_parms [ "batch_size" ],
-data_feature = None ,
-data_attribute = None ,
-attribute_cols_metadata = dp_metadata [ "attribute_cols_metadata" ],
-sample_len = dp_network_parms [ "sample_len" ],
-generator = generator ,
-discriminator = discriminator ,
-rounds = dp_network_parms [ "rounds" ],
-attr_discriminator = attr_discriminator ,
-d_gp_coe = dp_network_parms [ "d_gp_coe" ],
-attr_d_gp_coe = dp_network_parms [ "attr_d_gp_coe" ],
-g_attr_d_coe = dp_network_parms [ "g_attr_d_coe" ],
-num_packing = dp_network_parms [ "num_packing" ],
-attribute_latent_dim = dp_network_parms [ "attribute_latent_dim" ],
-feature_latent_dim = dp_network_parms [ "feature_latent_dim" ],
-fix_feature_network = dp_network_parms [ "fix_feature_network" ],
-g_lr = dp_network_parms [ "g_lr" ],
-g_beta1 = dp_network_parms [ "g_beta1" ],
-d_lr = dp_network_parms [ "d_lr" ],
-d_beta1 = dp_network_parms [ "d_beta1" ],
-attr_d_lr = dp_network_parms [ "attr_d_lr" ],
-attr_d_beta1 = dp_network_parms [ "attr_d_beta1" ])
-dp_model . _gan_model . sample_time = dp_network_parms [ "sample_time" ]
-dp_model . _gan_model . sample_feature_dim = dp_network_parms [ "sample_feature_dim" ]
-dp_model . _gan_model . sample_attribute_dim = dp_network_parms [ "sample_attribute_dim" ]
-dp_model . _gan_model . sample_real_attribute_dim = dp_network_parms [ "sample_real_attribute_dim" ]
-dp_model . _gan_model . build ()
-saver = tf . compat . v1 . train . Saver ()
-saver . restore ( sess , tf . compat . v1 . train . latest_checkpoint ( path ))
-dp_model . _tf_session = sess
-return dp_model
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols = None , cat_cols = None )
-
-
-
-
-
Fits the DoppelGANger model.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- DataFrame
-
-
-
A pandas DataFrame with the data to be synthesized.
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
DoppelGANger training arguments.
-
-
- required
-
-
-
- num_cols
- list[str] | None
-
-
-
-
List of columns to be handled as numerical
-
-
- None
-
-
-
- cat_cols
- list[str] | None
-
-
-
-
List of columns to be handled as categorical
-
-
- None
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/doppelganger/model.py
- def fit ( self , data : DataFrame ,
-train_arguments : TrainParameters ,
-num_cols : list [ str ] | None = None ,
-cat_cols : list [ str ] | None = None ):
- """
- Fits the DoppelGANger model.
- Args:
- data: A pandas DataFrame with the data to be synthesized.
- train_arguments: DoppelGANger training arguments.
- num_cols: List of columns to be handled as numerical
- cat_cols: List of columns to be handled as categorical
- """
-super () . fit ( data = data , num_cols = num_cols , cat_cols = cat_cols , train_arguments = train_arguments )
-self . _sequence_length = train_arguments . sequence_length
-self . _sample_length = train_arguments . sample_length
-self . _rounds = train_arguments . rounds
-if data . shape [ 0 ] % self . _sequence_length != 0 :
-raise ValueError ( "The number of samples must be a multiple of the sequence length." )
-if self . _sequence_length % self . _sample_length != 0 :
-raise ValueError ( "The sequence length must be a multiple of the sample length." )
-data_features , data_attributes = self . processor . transform ( data )
-measurement_cols_metadata = self . processor . measurement_cols_metadata
-attribute_cols_metadata = self . processor . attribute_cols_metadata
-generator = DoppelGANgerGenerator (
-feed_back = False ,
-noise = True ,
-use_tanh = self . use_tanh ,
-measurement_cols_metadata = measurement_cols_metadata ,
-attribute_cols_metadata = attribute_cols_metadata ,
-sample_len = self . _sample_length )
-discriminator = Discriminator ()
-attr_discriminator = AttrDiscriminator ()
-self . _tf_session = tf . compat . v1 . Session ()
-with self . _tf_session . as_default () as sess :
-self . _gan_model = DoppelGANgerNetwork (
-sess = sess ,
-epoch = train_arguments . epochs ,
-batch_size = self . batch_size ,
-data_feature = data_features ,
-data_attribute = data_attributes ,
-attribute_cols_metadata = attribute_cols_metadata ,
-sample_len = self . _sample_length ,
-generator = generator ,
-discriminator = discriminator ,
-rounds = self . _rounds ,
-attr_discriminator = attr_discriminator ,
-d_gp_coe = self . gp_lambda ,
-attr_d_gp_coe = self . gp_lambda ,
-g_attr_d_coe = self . gp_lambda ,
-num_packing = self . pac ,
-attribute_latent_dim = self . latent_dim ,
-feature_latent_dim = self . latent_dim ,
-fix_feature_network = False ,
-g_lr = self . g_lr ,
-g_beta1 = self . beta_1 ,
-d_lr = self . d_lr ,
-d_beta1 = self . beta_1 ,
-attr_d_lr = self . d_lr ,
-attr_d_beta1 = self . beta_1 )
-self . _gan_model . build ()
-self . _gan_model . train ()
-
-
-
-
-
-
-
-
-
-
-
- load ( path )
- staticmethod
-
-
-
-
-
-
-
Load the DoppelGANger model from a directory.
-Only the required components to sample new data are loaded.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- class_dict
-
-
-
-
Path of the directory where the files were saved.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/doppelganger/model.py
- @staticmethod
-def load ( path ):
- """
- Load the DoppelGANger model from a directory.
- Only the required components to sample new data are loaded.
- Args:
- class_dict: Path of the directory where the files were saved.
- """
-dp_model = DoppelGANger ( ModelParameters ())
-dp_network_parms = load ( os . path . join ( path , "doppelganger_network.pkl" ))
-dp_metadata = load ( os . path . join ( path , "doppelganger_metadata.pkl" ))
-dp_model . processor = DoppelGANgerProcessor ()
-dp_model . processor . __dict__ = dp_metadata [ "processor" ]
-dp_model . _sequence_length = dp_metadata [ "_sequence_length" ]
-dp_model . _sample_length = dp_metadata [ "_sample_length" ]
-generator = DoppelGANgerGenerator (
-feed_back = False ,
-noise = True ,
-measurement_cols_metadata = dp_metadata [ "measurement_cols_metadata" ],
-attribute_cols_metadata = dp_metadata [ "attribute_cols_metadata" ],
-sample_len = dp_network_parms [ "sample_len" ])
-discriminator = Discriminator ()
-attr_discriminator = AttrDiscriminator ()
-with tf . compat . v1 . Session () . as_default () as sess :
-dp_model . _gan_model = DoppelGANgerNetwork (
-sess = sess ,
-epoch = dp_network_parms [ "epoch" ],
-batch_size = dp_network_parms [ "batch_size" ],
-data_feature = None ,
-data_attribute = None ,
-attribute_cols_metadata = dp_metadata [ "attribute_cols_metadata" ],
-sample_len = dp_network_parms [ "sample_len" ],
-generator = generator ,
-discriminator = discriminator ,
-rounds = dp_network_parms [ "rounds" ],
-attr_discriminator = attr_discriminator ,
-d_gp_coe = dp_network_parms [ "d_gp_coe" ],
-attr_d_gp_coe = dp_network_parms [ "attr_d_gp_coe" ],
-g_attr_d_coe = dp_network_parms [ "g_attr_d_coe" ],
-num_packing = dp_network_parms [ "num_packing" ],
-attribute_latent_dim = dp_network_parms [ "attribute_latent_dim" ],
-feature_latent_dim = dp_network_parms [ "feature_latent_dim" ],
-fix_feature_network = dp_network_parms [ "fix_feature_network" ],
-g_lr = dp_network_parms [ "g_lr" ],
-g_beta1 = dp_network_parms [ "g_beta1" ],
-d_lr = dp_network_parms [ "d_lr" ],
-d_beta1 = dp_network_parms [ "d_beta1" ],
-attr_d_lr = dp_network_parms [ "attr_d_lr" ],
-attr_d_beta1 = dp_network_parms [ "attr_d_beta1" ])
-dp_model . _gan_model . sample_time = dp_network_parms [ "sample_time" ]
-dp_model . _gan_model . sample_feature_dim = dp_network_parms [ "sample_feature_dim" ]
-dp_model . _gan_model . sample_attribute_dim = dp_network_parms [ "sample_attribute_dim" ]
-dp_model . _gan_model . sample_real_attribute_dim = dp_network_parms [ "sample_real_attribute_dim" ]
-dp_model . _gan_model . build ()
-saver = tf . compat . v1 . train . Saver ()
-saver . restore ( sess , tf . compat . v1 . train . latest_checkpoint ( path ))
-dp_model . _tf_session = sess
-return dp_model
-
-
-
-
-
-
-
-
-
-
-
- sample ( n_samples )
-
-
-
-
-
Samples new data from the DoppelGANger.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- n_samples
- int
-
-
-
-
Number of samples to be generated.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/doppelganger/model.py
- def sample ( self , n_samples : int ):
- """
- Samples new data from the DoppelGANger.
- Args:
- n_samples: Number of samples to be generated.
- """
-if n_samples <= 0 :
-raise ValueError ( "Invalid number of samples." )
-real_attribute_input_noise = self . _gan_model . gen_attribute_input_noise ( n_samples )
-addi_attribute_input_noise = self . _gan_model . gen_attribute_input_noise ( n_samples )
-length = int ( self . _sequence_length / self . _sample_length )
-feature_input_noise = self . _gan_model . gen_feature_input_noise ( n_samples , length = length )
-input_data = self . _gan_model . gen_feature_input_data_free ( n_samples )
-with self . _tf_session . as_default () as sess :
-self . _gan_model . sess = sess
-data_features , data_attributes , gen_flags , _ = self . _gan_model . sample_from (
-real_attribute_input_noise , addi_attribute_input_noise ,
-feature_input_noise , input_data )
-return self . processor . inverse_transform ( data_features , data_attributes , gen_flags )
-
-
-
-
-
-
-
-
-
-
-
- save ( path )
-
-
-
-
-
Save the DoppelGANger model in a directory.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- path
-
-
-
-
Path of the directory where the files will be saved.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/doppelganger/model.py
- def save ( self , path ):
- """
- Save the DoppelGANger model in a directory.
- Args:
- path: Path of the directory where the files will be saved.
- """
-saver = tf . compat . v1 . train . Saver ()
-with self . _tf_session . as_default () as sess :
-saver . save ( sess , os . path . join ( path , "doppelganger" ), write_meta_graph = False )
-self . _gan_model . save ( os . path . join ( path , "doppelganger_network.pkl" ))
-dump ({
-"processor" : self . processor . __dict__ ,
-"measurement_cols_metadata" : self . processor . measurement_cols_metadata ,
-"attribute_cols_metadata" : self . processor . attribute_cols_metadata ,
-"_sequence_length" : self . _sequence_length ,
-"_sample_length" : self . _sample_length
-}, os . path . join ( path , "doppelganger_metadata.pkl" ))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
TimeGAN - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- TimeGAN
-
-
-
-
-
-
-
-
- Bases: BaseGANModel
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/timegan/model.py
- class TimeGAN ( BaseGANModel ):
-__MODEL__ = 'TimeGAN'
-def __init__ ( self , model_parameters : ModelParameters ):
-super () . __init__ ( model_parameters )
-self . seq_len = None
-self . n_seq = None
-self . hidden_dim = model_parameters . latent_dim
-self . gamma = model_parameters . gamma
-self . num_cols = None
-def fit ( self , data : DataFrame ,
-train_arguments : TrainParameters ,
-num_cols : list [ str ] | None = None ,
-cat_cols : list [ str ] | None = None ):
- """
- Fits the TimeGAN model.
- Args:
- data: A pandas DataFrame with the data to be synthesized.
- train_arguments: TimeGAN training arguments.
- num_cols: List of columns to be handled as numerical
- cat_cols: List of columns to be handled as categorical
- """
-super () . fit ( data = data , num_cols = num_cols , cat_cols = cat_cols , train_arguments = train_arguments )
-if cat_cols :
-raise NotImplementedError ( "TimeGAN does not support categorical features." )
-self . num_cols = num_cols
-self . seq_len = train_arguments . sequence_length
-self . n_seq = train_arguments . number_sequences
-processed_data = real_data_loading ( data [ self . num_cols ] . values , seq_len = self . seq_len )
-self . train ( data = processed_data , train_steps = train_arguments . epochs )
-def sample ( self , n_samples : int ):
- """
- Samples new data from the TimeGAN.
- Args:
- n_samples: Number of samples to be generated.
- """
-Z_ = next ( self . get_batch_noise ( size = n_samples ))
-records = self . generator ( Z_ )
-data = []
-for i in range ( records . shape [ 0 ]):
-data . append ( DataFrame ( records [ i ], columns = self . num_cols ))
-return data
-def define_gan ( self ):
-self . generator_aux = Generator ( self . hidden_dim ) . build ()
-self . supervisor = Supervisor ( self . hidden_dim ) . build ()
-self . discriminator = Discriminator ( self . hidden_dim ) . build ()
-self . recovery = Recovery ( self . hidden_dim , self . n_seq ) . build ()
-self . embedder = Embedder ( self . hidden_dim ) . build ()
-X = Input ( shape = [ self . seq_len , self . n_seq ], batch_size = self . batch_size , name = 'RealData' )
-Z = Input ( shape = [ self . seq_len , self . n_seq ], batch_size = self . batch_size , name = 'RandomNoise' )
-#--------------------------------
-# Building the AutoEncoder
-#--------------------------------
-H = self . embedder ( X )
-X_tilde = self . recovery ( H )
-self . autoencoder = Model ( inputs = X , outputs = X_tilde )
-#---------------------------------
-# Adversarial Supervise Architecture
-#---------------------------------
-E_Hat = self . generator_aux ( Z )
-H_hat = self . supervisor ( E_Hat )
-Y_fake = self . discriminator ( H_hat )
-self . adversarial_supervised = Model ( inputs = Z ,
-outputs = Y_fake ,
-name = 'AdversarialSupervised' )
-#---------------------------------
-# Adversarial architecture in latent space
-#---------------------------------
-Y_fake_e = self . discriminator ( E_Hat )
-self . adversarial_embedded = Model ( inputs = Z ,
-outputs = Y_fake_e ,
-name = 'AdversarialEmbedded' )
-# ---------------------------------
-# Synthetic data generation
-# ---------------------------------
-X_hat = self . recovery ( H_hat )
-self . generator = Model ( inputs = Z ,
-outputs = X_hat ,
-name = 'FinalGenerator' )
-# --------------------------------
-# Final discriminator model
-# --------------------------------
-Y_real = self . discriminator ( H )
-self . discriminator_model = Model ( inputs = X ,
-outputs = Y_real ,
-name = "RealDiscriminator" )
-# ----------------------------
-# Define the loss functions
-# ----------------------------
-self . _mse = MeanSquaredError ()
-self . _bce = BinaryCrossentropy ()
-@function
-def train_autoencoder ( self , x , opt ):
-with GradientTape () as tape :
-x_tilde = self . autoencoder ( x )
-embedding_loss_t0 = self . _mse ( x , x_tilde )
-e_loss_0 = 10 * sqrt ( embedding_loss_t0 )
-var_list = self . embedder . trainable_variables + self . recovery . trainable_variables
-gradients = tape . gradient ( e_loss_0 , var_list )
-opt . apply_gradients ( zip ( gradients , var_list ))
-return sqrt ( embedding_loss_t0 )
-@function
-def train_supervisor ( self , x , opt ):
-with GradientTape () as tape :
-h = self . embedder ( x )
-h_hat_supervised = self . supervisor ( h )
-generator_loss_supervised = self . _mse ( h [:, 1 :, :], h_hat_supervised [:, : - 1 , :])
-var_list = self . supervisor . trainable_variables + self . generator . trainable_variables
-gradients = tape . gradient ( generator_loss_supervised , var_list )
-apply_grads = [( grad , var ) for ( grad , var ) in zip ( gradients , var_list ) if grad is not None ]
-opt . apply_gradients ( apply_grads )
-return generator_loss_supervised
-@function
-def train_embedder ( self , x , opt ):
-with GradientTape () as tape :
-# Supervised Loss
-h = self . embedder ( x )
-h_hat_supervised = self . supervisor ( h )
-generator_loss_supervised = self . _mse ( h [:, 1 :, :], h_hat_supervised [:, : - 1 , :])
-# Reconstruction Loss
-x_tilde = self . autoencoder ( x )
-embedding_loss_t0 = self . _mse ( x , x_tilde )
-e_loss = 10 * sqrt ( embedding_loss_t0 ) + 0.1 * generator_loss_supervised
-var_list = self . embedder . trainable_variables + self . recovery . trainable_variables
-gradients = tape . gradient ( e_loss , var_list )
-opt . apply_gradients ( zip ( gradients , var_list ))
-return sqrt ( embedding_loss_t0 )
-def discriminator_loss ( self , x , z ):
-# Loss on false negatives
-y_real = self . discriminator_model ( x )
-discriminator_loss_real = self . _bce ( y_true = ones_like ( y_real ),
-y_pred = y_real )
-# Loss on false positives
-y_fake = self . adversarial_supervised ( z )
-discriminator_loss_fake = self . _bce ( y_true = zeros_like ( y_fake ),
-y_pred = y_fake )
-y_fake_e = self . adversarial_embedded ( z )
-discriminator_loss_fake_e = self . _bce ( y_true = zeros_like ( y_fake_e ),
-y_pred = y_fake_e )
-return ( discriminator_loss_real +
-discriminator_loss_fake +
-self . gamma * discriminator_loss_fake_e )
-@staticmethod
-def calc_generator_moments_loss ( y_true , y_pred ):
-y_true_mean , y_true_var = nn . moments ( x = y_true , axes = [ 0 ])
-y_pred_mean , y_pred_var = nn . moments ( x = y_pred , axes = [ 0 ])
-g_loss_mean = reduce_mean ( abs ( y_true_mean - y_pred_mean ))
-g_loss_var = reduce_mean ( abs ( sqrt ( y_true_var + 1e-6 ) - sqrt ( y_pred_var + 1e-6 )))
-return g_loss_mean + g_loss_var
-@function
-def train_generator ( self , x , z , opt ):
-with GradientTape () as tape :
-y_fake = self . adversarial_supervised ( z )
-generator_loss_unsupervised = self . _bce ( y_true = ones_like ( y_fake ),
-y_pred = y_fake )
-y_fake_e = self . adversarial_embedded ( z )
-generator_loss_unsupervised_e = self . _bce ( y_true = ones_like ( y_fake_e ),
-y_pred = y_fake_e )
-h = self . embedder ( x )
-h_hat_supervised = self . supervisor ( h )
-generator_loss_supervised = self . _mse ( h [:, 1 :, :], h_hat_supervised [:, : - 1 , :])
-x_hat = self . generator ( z )
-generator_moment_loss = self . calc_generator_moments_loss ( x , x_hat )
-generator_loss = ( generator_loss_unsupervised +
-generator_loss_unsupervised_e +
-100 * sqrt ( generator_loss_supervised ) +
-100 * generator_moment_loss )
-var_list = self . generator_aux . trainable_variables + self . supervisor . trainable_variables
-gradients = tape . gradient ( generator_loss , var_list )
-opt . apply_gradients ( zip ( gradients , var_list ))
-return generator_loss_unsupervised , generator_loss_supervised , generator_moment_loss
-@function
-def train_discriminator ( self , x , z , opt ):
-with GradientTape () as tape :
-discriminator_loss = self . discriminator_loss ( x , z )
-var_list = self . discriminator . trainable_variables
-gradients = tape . gradient ( discriminator_loss , var_list )
-opt . apply_gradients ( zip ( gradients , var_list ))
-return discriminator_loss
-def get_batch_data ( self , data , n_windows ):
-data = convert_to_tensor ( data , dtype = float32 )
-return iter ( tfdata . Dataset . from_tensor_slices ( data )
-. shuffle ( buffer_size = n_windows )
-. batch ( self . batch_size ) . repeat ())
-def _generate_noise ( self ):
-while True :
-yield np . random . uniform ( low = 0 , high = 1 , size = ( self . seq_len , self . n_seq ))
-def get_batch_noise ( self , size = None ):
-return iter ( tfdata . Dataset . from_generator ( self . _generate_noise , output_types = float32 )
-. batch ( self . batch_size if size is None else size )
-. repeat ())
-def train ( self , data , train_steps ):
-# Assemble the model
-self . define_gan ()
-## Embedding network training
-autoencoder_opt = Adam ( learning_rate = self . g_lr )
-for _ in tqdm ( range ( train_steps ), desc = 'Emddeding network training' ):
-X_ = next ( self . get_batch_data ( data , n_windows = len ( data )))
-step_e_loss_t0 = self . train_autoencoder ( X_ , autoencoder_opt )
-## Supervised Network training
-supervisor_opt = Adam ( learning_rate = self . g_lr )
-for _ in tqdm ( range ( train_steps ), desc = 'Supervised network training' ):
-X_ = next ( self . get_batch_data ( data , n_windows = len ( data )))
-step_g_loss_s = self . train_supervisor ( X_ , supervisor_opt )
-## Joint training
-generator_opt = Adam ( learning_rate = self . g_lr )
-embedder_opt = Adam ( learning_rate = self . g_lr )
-discriminator_opt = Adam ( learning_rate = self . d_lr )
-step_g_loss_u = step_g_loss_s = step_g_loss_v = step_e_loss_t0 = step_d_loss = 0
-for _ in tqdm ( range ( train_steps ), desc = 'Joint networks training' ):
-#Train the generator (k times as often as the discriminator)
-# Here k=2
-for _ in range ( 2 ):
-X_ = next ( self . get_batch_data ( data , n_windows = len ( data )))
-Z_ = next ( self . get_batch_noise ())
-# --------------------------
-# Train the generator
-# --------------------------
-step_g_loss_u , step_g_loss_s , step_g_loss_v = self . train_generator ( X_ , Z_ , generator_opt )
-# --------------------------
-# Train the embedder
-# --------------------------
-step_e_loss_t0 = self . train_embedder ( X_ , embedder_opt )
-X_ = next ( self . get_batch_data ( data , n_windows = len ( data )))
-Z_ = next ( self . get_batch_noise ())
-step_d_loss = self . discriminator_loss ( X_ , Z_ )
-if step_d_loss > 0.15 :
-step_d_loss = self . train_discriminator ( X_ , Z_ , discriminator_opt )
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- fit ( data , train_arguments , num_cols = None , cat_cols = None )
-
-
-
-
-
Fits the TimeGAN model.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- data
- DataFrame
-
-
-
A pandas DataFrame with the data to be synthesized.
-
-
- required
-
-
-
- train_arguments
- TrainParameters
-
-
-
TimeGAN training arguments.
-
-
- required
-
-
-
- num_cols
- list[str] | None
-
-
-
-
List of columns to be handled as numerical
-
-
- None
-
-
-
- cat_cols
- list[str] | None
-
-
-
-
List of columns to be handled as categorical
-
-
- None
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/timegan/model.py
- def fit ( self , data : DataFrame ,
-train_arguments : TrainParameters ,
-num_cols : list [ str ] | None = None ,
-cat_cols : list [ str ] | None = None ):
- """
- Fits the TimeGAN model.
- Args:
- data: A pandas DataFrame with the data to be synthesized.
- train_arguments: TimeGAN training arguments.
- num_cols: List of columns to be handled as numerical
- cat_cols: List of columns to be handled as categorical
- """
-super () . fit ( data = data , num_cols = num_cols , cat_cols = cat_cols , train_arguments = train_arguments )
-if cat_cols :
-raise NotImplementedError ( "TimeGAN does not support categorical features." )
-self . num_cols = num_cols
-self . seq_len = train_arguments . sequence_length
-self . n_seq = train_arguments . number_sequences
-processed_data = real_data_loading ( data [ self . num_cols ] . values , seq_len = self . seq_len )
-self . train ( data = processed_data , train_steps = train_arguments . epochs )
-
-
-
-
-
-
-
-
-
-
-
- sample ( n_samples )
-
-
-
-
-
Samples new data from the TimeGAN.
-
-
-
-
Parameters:
-
-
-
- Name
- Type
- Description
- Default
-
-
-
-
- n_samples
- int
-
-
-
-
Number of samples to be generated.
-
-
- required
-
-
-
-
-
-
- Source code in ydata_synthetic/synthesizers/timeseries/timegan/model.py
- def sample ( self , n_samples : int ):
- """
- Samples new data from the TimeGAN.
- Args:
- n_samples: Number of samples to be generated.
- """
-Z_ = next ( self . get_batch_noise ( size = n_samples ))
-records = self . generator ( Z_ )
-data = []
-for i in range ( records . shape [ 0 ]):
-data . append ( DataFrame ( records [ i ], columns = self . num_cols ))
-return data
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Changelog - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Analytics - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Analytics & Telemetry
-Overview
-ydata-synthetic is a powerful library designed to generate synthetic data.
-As part of our ongoing efforts to improve user experience and functionality, ydata-synthetic
-includes a telemetry feature. This feature collects anonymous usage data, helping us understand
-how the library is used and identify areas for improvement.
-The primary goal of collecting telemetry data is to:
-
-Enhance the functionality and performance of the ydata-synthetic library
-Prioritize new features based on user engagement
-Identify common issues and bugs to improve overall user experience
-
-Data Collected
-The telemetry system collects non-personal, anonymous information such as:
-
-Python version
-ydata-synthetic versionFrequency of use of ydata-synthetic features
-Errors or exceptions thrown within the library
-
-Disabling usage analytics
-We respect your choice to not participate in our telemetry collection.
-If you prefer to disable telemetry, you can do so by setting an environment
-variable on your system. Disabling telemetry will not affect the functionality
-of the ydata-profiling library, except for the ability to contribute to its usage analytics.
-Set an Environment Variable
-In your notebook or script make sure to set YDATA_SYNTHETIC_NO_ANALYTICS
-environment variable to True.
-import os
-os . environ [ 'YDATA_SYNTHETIC_NO_ANALYTICS' ] = 'True'
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Contribution Guidelines - YData-Synthetic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Contribution Guidelines
-
-Contribute
-You are always welcome to contribute to this incredible ecosystem for synthetic data generation. We have several areas in that we are always looking for an extra pair of hands to help us get things going:
-
-Documentation: we all love it, but keeping it up-to-date can be challenging! But guess what? This is also the fastest way for you to get to know ydata-synthetic and start contributing! Even if it is missing, like a new example that could help the community go from zero to hero with synthetic data!Getting started: Issues that use this tag are usually the most friendly for someone that has just begun the journey into open-source contributions. So don't be shy; assign the task to you and introduce yourself in the GitHub issue! We will be there to guide you.
-But we always look for contributions that go beyond documentation and fixes
-
-Synthetic data for NLP: If you are an expert or just someone that would like to dive into this topic, we welcome you to participate in a small project around the generation of synthetic text.Synthetic data for images: if you are a computer vision and you like to share, feel free to add some examples for images!Any other research around synthetic data you would like to share with the community is more than welcome! If you want your research to be part of a rich ecosystem, open a PR!
-
-PR name and convention
-For the commit messages, if you could please read this commit-message guidelines or conventional commit messages and follow these conventions, it would be awesome.
-Issues and where I can find more info about contributing
-If you need help figuring out where to start or want to learn what other contributors are doing, go to the Data-Centric AI community Discord channel and introduce yourself in the #ydata-synthetic channel.
-If you can't find an issue that interests you and wants to add an improvement or change, create a new one.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Synthetic Data Generation
-
-
-
-
-
-
-
-
-
-
-
-
-
Integrations
@@ -258,22 +240,6 @@
-
-
-
-
-
-
-
-
-
-
- Reference
-
-
-
-
-
+
@@ -1554,41 +948,57 @@ 
