Implement SECRET token (#571)

* JIRA-WDT-370 Implement @@secret token with basic directory structure

* JIRA-WDT-370 Implement @@secret token lookup using WDT_MODEL_SECRETS_NAME_DIR_PAIRS

* JIRA-WDT-370 Implement @@secret token lookup using WDT_MODEL_SECRETS_NAME_DIR_PAIRS

Author: rakillen

README.md

@@ -176,9 +176,9 @@ As the example above shows, the `SecurityConfiguration` element has no named sub
 
 The model allows the use of tokens that are substituted with text values as the model is processed. This section describes several types of tokens.
 
-**Variable placeholders** are declared with the syntax `@@PROP:<variable>@@`. This type of token represents a value that is resolved at runtime using a variables file in a standard Java properties file format.  Variables can be used for any value and for some names.  For example, to automate standing up an environment with one or more applications in the Oracle Java Cloud Service, service provisioning does not allow the provisioning script to specify the server names.  For example, if the application being deployed immediately following provisioning needs to tweak the Server Start arguments to specify a Java system property, the model can use a variable placeholder in place of the server name and populate the variable file with the provisioned server names dynamically between provisioning and application deployment.
+**Variable tokens** are declared with the syntax `@@PROP:<variable>@@`. This type of token represents a value that is resolved at runtime using a variables file in a standard Java properties file format.  Variables can be used for any value and for some names.  For example, to automate standing up an environment with one or more applications in the Oracle Java Cloud Service, service provisioning does not allow the provisioning script to specify the server names.  For example, if the application being deployed immediately following provisioning needs to tweak the Server Start arguments to specify a Java system property, the model can use a variable placeholder in place of the server name and populate the variable file with the provisioned server names dynamically between provisioning and application deployment.
 
-**File placeholders** are declared with the syntax `@@FILE:<filename>@@`. This type of token is similar to a variable placeholder, but the token references a single value that is read from the specified file. For example, the model may reference a password attribute as follows:
+**File tokens** are declared with the syntax `@@FILE:<filename>@@`. This type of token is similar to a variable token, but it references a single value that is read from the specified file. For example, the model may reference a password attribute as follows:
 ```yaml
 PasswordEncrypted: '@@FILE:/home/me/dbcs1.txt@@'
 ```
@@ -194,6 +194,27 @@ PasswordEncrypted: '@@FILE:/dir/@@PROP:name@@.txt@@'
 PasswordEncrypted: '@@FILE:@@ORACLE_HOME@@/dir/name.txt@@'
 ```
 
+**Environment tokens** are declared with the syntax `@@ENV:<name>@@`. This type of token is resolved by looking up the system environment variable `<name>`, and substituting that value for the token.
+
+**Secret tokens** are declared with the syntax `@@SECRET:<name>:<key>@@`. This type of token is resolved by determining the location of a Kubernetes secret file, and reading the first line from that file. That line is substituted for the token.
+
+There are two methods for deriving the location of the Kubernetes secret file. The first method involves using one or more configured root directories, and looking for the secret file in the path `<root-directory>/<name>/<key>`.
+
+The root directories are configured as a comma-separated list of directories, using the environment variable `WDT_MODEL_SECRETS_DIRS`. For example, if `WDT_MODEL_SECRETS_DIRS` is set to `/etc/my-secrets,/etc/your-secrets`, then the token `@@SECRET:secrets:the-secret@@` will search the following locations:
+```
+/etc/my-secrets/secrets/the-secret
+/etc/your-secrets/secrets/the-secret
+``` 
+If either of these files is found, the secret is read from that file and substituted in the model.
+
+The second method for locating the Kubernetes secret file is to use the environment variable `WDT_MODEL_SECRETS_NAME_DIR_PAIRS` to map `<name>` values to specific directory locations. For example, if `WDT_MODEL_SECRETS_NAME_DIR_PAIRS` is set to `my-root=/etc/my-secrets,your-root=/etc/your-secrets`, then the token `@@SECRET:your-root:the-secret@@` will look for the secrets file at: 
+```
+/etc/your-secrets/the-secret
+```
+If the `<name>` value has a corresponding mapped directory in `WDT_MODEL_SECRETS_NAME_DIR_PAIRS`, then that directory will take precedence over any roots specified in `WDT_MODEL_SECRETS_DIRS`. 
+
+NOTE: It is important that the secrets directories contain only secrets files, because those files are examined to create a list of available name/key pairs.  
+
 **Path tokens** are tokens that reference known values, and can be used to make the model more portable. For example, a model may reference a WebLogic library source path as:
 ```yaml
 SourcePath: '@@WL_HOME@@/common/deployable-libraries/jsf-2.0.war'

core/src/main/python/validate.py

@@ -11,6 +11,7 @@
 
 from oracle.weblogic.deploy.util import CLAException
 from oracle.weblogic.deploy.util import TranslateException
+from oracle.weblogic.deploy.util import VariableException
 from oracle.weblogic.deploy.util import WebLogicDeployToolingVersion
 from oracle.weblogic.deploy.validate import ValidateException
 
@@ -186,7 +187,7 @@ def __perform_model_file_validation(model_file_name, model_context):
 
         model_validator.validate_in_standalone_mode(model_dictionary, variable_map,
                                                     model_context.get_archive_file_name())
-    except TranslateException, te:
+    except (TranslateException, VariableException), te:
         __logger.severe('WLSDPLY-20009', _program_name, model_file_name, te.getLocalizedMessage(),
                         error=te, class_name=_class_name, method_name=_method_name)
         ex = exception_helper.create_validate_exception(te.getLocalizedMessage(), error=te)

core/src/main/python/wlsdeploy/util/variables.py

@@ -20,14 +20,22 @@
 from wlsdeploy.util import path_utils
 from wlsdeploy.exception import exception_helper
 from wlsdeploy.logging import platform_logger
+from wlsdeploy.util import dictionary_utils
 
 _class_name = "variables"
 _logger = platform_logger.PlatformLogger('wlsdeploy.variables')
 _file_variable_pattern = re.compile("@@FILE:[\w.\\\/:-]+@@")
 _property_pattern = re.compile("(@@PROP:([\\w.-]+)@@)")
 _environment_pattern = re.compile("(@@ENV:([\\w.-]+)@@)")
+_secret_pattern = re.compile("(@@SECRET:([\\w.-]+):([\\w.-]+)@@)")
 _file_nested_variable_pattern = re.compile("@@FILE:@@[\w]+@@[\w.\\\/:-]+@@")
 
+_secret_dirs_variable = "WDT_MODEL_SECRETS_DIRS"
+_secret_dirs_default = "/weblogic-operator/config-overrides-secrets"
+_secret_dir_pairs_variable="WDT_MODEL_SECRETS_NAME_DIR_PAIRS"
+
+_secret_token_map = None
+
 
 def load_variables(file_path):
     """
@@ -207,10 +215,10 @@ def _process_node(nodes, variables, model_context):
 
 def _substitute(text, variables, model_context):
     """
-    Substitute the variable placeholders with the variable value.
-    :param text: the text to process for variable placeholders
+    Substitute token placeholders with their derived values.
+    :param text: the text to process for token placeholders
     :param variables: the variables to use
-    :param model_context: used to resolve variables in file paths
+    :param model_context: used to determine the validation method (strict, lax, etc.)
     :return: the replaced text
     """
     method_name = '_substitute'
@@ -223,14 +231,8 @@ def _substitute(text, variables, model_context):
         for token, key in matches:
             # log, or throw an exception if key is not found.
             if key not in variables:
-                if model_context.get_validation_method() == 'strict':
-                    _logger.severe('WLSDPLY-01732', key, class_name=_class_name, method_name=method_name)
-                    ex = exception_helper.create_variable_exception('WLSDPLY-01732', key)
-                    _logger.throwing(ex, class_name=_class_name, method_name=method_name)
-                    raise ex
-                else:
-                    _logger.info('WLSDPLY-01732', key, class_name=_class_name, method_name=method_name)
-                    continue
+                _report_token_issue('WLSDPLY-01732', method_name, model_context, key)
+                continue
 
             value = variables[key]
             text = text.replace(token, value)
@@ -240,18 +242,24 @@ def _substitute(text, variables, model_context):
         for token, key in matches:
             # log, or throw an exception if key is not found.
             if not os.environ.has_key(key):
-                if model_context.get_validation_method() == 'strict':
-                    _logger.severe('WLSDPLY-01737', key, class_name=_class_name, method_name=method_name)
-                    ex = exception_helper.create_variable_exception('WLSDPLY-01737', key)
-                    _logger.throwing(ex, class_name=_class_name, method_name=method_name)
-                    raise ex
-                else:
-                    _logger.info('WLSDPLY-01737', key, class_name=_class_name, method_name=method_name)
-                    continue
+                _report_token_issue('WLSDPLY-01737', method_name, model_context, key)
+                continue
 
             value = os.environ.get(key)
             text = text.replace(token, value)
 
+        # check secret variables before @@FILE:/dir/@@SECRET:name:key@@.txt@@
+        matches = _secret_pattern.findall(text)
+        for token, name, key in matches:
+            value = _resolve_secret_token(name, key, model_context)
+            if value is None:
+                secret_token = name + ':' + key
+                known_tokens = _list_known_secret_tokens()
+                _report_token_issue('WLSDPLY-01739', method_name, model_context, secret_token, known_tokens)
+                continue
+
+            text = text.replace(token, value)
+
         tokens = _file_variable_pattern.findall(text)
         if tokens:
             for token in tokens:
@@ -285,16 +293,8 @@ def _read_value_from_file(file_path, model_context):
         line = file_reader.readLine()
         file_reader.close()
     except IOException, e:
-        if model_context.get_validation_method() == 'strict':
-            _logger.severe('WLSDPLY-01733', file_path, e.getLocalizedMessage(), class_name=_class_name,
-                           method_name=method_name)
-            ex = exception_helper.create_variable_exception('WLSDPLY-01733', file_path, e.getLocalizedMessage(), error=e)
-            _logger.throwing(ex, class_name=_class_name, method_name=method_name)
-            raise ex
-        else:
-            _logger.info('WLSDPLY-01733', file_path, e.getLocalizedMessage(), error=e, class_name=_class_name,
-                         method_name=method_name)
-            line = ''
+        _report_token_issue('WLSDPLY-01733', method_name, model_context, file_path, e.getLocalizedMessage())
+        line = ''
 
     if line is None:
         ex = exception_helper.create_variable_exception('WLSDPLY-01734', file_path)
@@ -304,6 +304,141 @@ def _read_value_from_file(file_path, model_context):
     return str(line).strip()
 
 
+def _resolve_secret_token(name, key, model_context):
+    """
+    Return the value associated with the specified secret name and key.
+    If the name and key are found in the directory map, return the associated value.
+    :param name: the name of the secret (a directory name or mapped name)
+    :param key: the name of the file containing the secret
+    :param model_context: used to determine the validation method (strict, lax, etc.)
+    :return: the secret value, or None if it is not found
+    """
+    method_name = '_resolve_secret_token'
+    global _secret_token_map
+
+    if _secret_token_map is None:
+        _init_secret_token_map(model_context)
+
+    secret_token = name + ':' + key
+    return dictionary_utils.get_element(_secret_token_map, secret_token)
+
+
+def _init_secret_token_map(model_context):
+    """
+    Initialize a global map of name/value tokens to secret values.
+    The map includes secrets found below the directories specified in WDT_MODEL_SECRETS_DIRS,
+    and in WDT_MODEL_SECRETS_NAME_DIR_PAIRS assignments.
+    :param model_context: used to determine the validation method (strict, lax, etc.)
+    """
+    method_name = '_init_secret_token_map'
+    global _secret_token_map
+
+    log_method = _logger.info
+    if model_context.get_validation_method() == 'strict':
+        log_method = _logger.warning
+
+    _secret_token_map = dict()
+
+    # add name/key pairs for files in sub-directories of directories in WDT_MODEL_SECRETS_DIRS.
+
+    locations = os.environ.get(_secret_dirs_variable, _secret_dirs_default)
+    for dir in locations.split(","):
+         if not os.path.isdir(dir):
+             # log at WARN or INFO, but no exception is thrown
+             log_method('WLSDPLY-01738', _secret_dirs_variable, dir, class_name=_class_name, method_name=method_name)
+             continue
+
+         for subdir_name in os.listdir(dir):
+             subdir_path = os.path.join(dir, subdir_name)
+             if os.path.isdir(subdir_path):
+                 _add_file_secrets_to_map(subdir_path, subdir_name, model_context)
+
+    # add name/key pairs for files in directories assigned in WDT_MODEL_SECRETS_NAME_DIR_PAIRS.
+    # these pairs will override if they were previously added as sub-directory pairs.
+
+    dir_pairs_text = os.environ.get(_secret_dir_pairs_variable, None)
+    if dir_pairs_text is not None:
+        dir_pairs = dir_pairs_text.split(',')
+        for dir_pair in dir_pairs:
+            result = dir_pair.split('=')
+            if len(result) != 2:
+                log_method('WLSDPLY-01735', _secret_dir_pairs_variable, dir_pair, class_name=_class_name,
+                           method_name=method_name)
+                continue
+
+            dir = result[1]
+            if not os.path.isdir(dir):
+                log_method('WLSDPLY-01738', _secret_dir_pairs_variable, dir, class_name=_class_name,
+                           method_name=method_name)
+                continue
+
+            name = result[0]
+            _add_file_secrets_to_map(dir, name, model_context)
+
+
+def _clear_secret_token_map():
+    """
+    Used by unit tests to force reload of map.
+    """
+    global _secret_token_map
+    _secret_token_map = None
+
+
+def _add_file_secrets_to_map(dir, name, model_context):
+    """
+    Add the secret from each file in the specified directory to the map.
+    :param dir: the directory to be examined
+    :param name: the name to be used in the map token
+    :param model_context: used to determine the validation method (strict, lax, etc.)
+    """
+    global _secret_token_map
+
+    for file_name in os.listdir(dir):
+        file_path = os.path.join(dir, file_name)
+        if os.path.isfile(file_path):
+            token = name + ":" + file_name
+            _secret_token_map[token] = _read_value_from_file(file_path, model_context)
+
+
+def _list_known_secret_tokens():
+    """
+    Returns a string representation of the available secret name/path tokens.
+    """
+    global _secret_token_map
+
+    keys = list(_secret_token_map.keys())
+    keys.sort()
+
+    ret = ''
+    for key in keys:
+        if ret != '':
+            ret += ', '
+        ret += "'" + key + "'"
+    return ret
+
+
+def _report_token_issue(message_key, method_name, model_context, *args):
+    """
+    Log a message at the level corresponding to the validation method (SEVERE for strict, INFO otherwise).
+    Throw a variable exception if the level is strict.
+    The lax validation method can be used to verify the model without resolving tokens.
+    :param message_key: the message key to be logged and used for exceptions
+    :param method_name: the name of the calling method for logging
+    :param model_context: used to determine the validation method
+    :param args: arguments for use in the message
+    """
+    log_method = _logger.info
+    if model_context.get_validation_method() == 'strict':
+        log_method = _logger.severe
+
+    log_method(message_key, class_name=_class_name, method_name=method_name, *args)
+
+    if model_context.get_validation_method() == 'strict':
+        ex = exception_helper.create_variable_exception(message_key, *args)
+        _logger.throwing(ex, class_name=_class_name, method_name=method_name)
+        raise ex
+
+
 def substitute_key(text, variables):
     """
     Substitute any @@PROP values in the text and return.

core/src/main/resources/oracle/weblogic/deploy/messages/wlsdeploy_rb.properties

@@ -291,9 +291,11 @@ WLSDPLY-01731=Variables updated by {0}
 WLSDPLY-01732=Variable {0} is not found in properties file
 WLSDPLY-01733=Variable file {0} cannot be read: {1}
 WLSDPLY-01734=No value in variable file {0}
-WLSDPLY-01735=Variable substitution for {0} is deprecated, use @@PROP:{1}@@
+WLSDPLY-01735=Assignment in {0} is not formatted as X=Y: {1}
 WLSDPLY-01736=Default variable file name {0}
 WLSDPLY-01737=Environment variable "{0}" was not found
+WLSDPLY-01738=Path in {0} is not a directory: {1}
+WLSDPLY-01739=Error: Could not resolve secret token "{0}". Known secrets include: {1}
 
 # wlsdeploy/util/weblogic_helper.py
 WLSDPLY-01740=Encryption failed: Unable to locate SerializedSystemIni

core/src/test/python/variables_test.py

@@ -111,6 +111,39 @@ def testEnvironmentVariableNotFound(self):
         else:
             self.fail('Test must raise VariableException when variable file is not found')
 
+    def testSecretToken(self):
+        """
+        Verify that the WDT_MODEL_SECRETS_DIRS variable can be used to find a secret.
+        Put two paths in the variable, the second is .../resources/secrets.
+        It should find the file .../resources/secrets/my-secrets/secret2, containing "mySecret2".
+        """
+        os.environ['WDT_MODEL_SECRETS_DIRS'] = "/noDir/noSubdir," + self._resources_dir + "/secrets"
+        model = {'domainInfo': {'AdminUserName': '@@SECRET:my-secrets:secret2@@'}}
+        variables._clear_secret_token_map()
+        variables.substitute(model, {}, self.model_context)
+        self.assertEqual(model['domainInfo']['AdminUserName'], 'mySecret2')
+
+    def testSecretTokenPairs(self):
+        """
+        Verify that the WDT_MODEL_SECRETS_NAME_DIR_PAIRS variable can be used to find a secret.
+        Put two path assignments in the variable, the second is dirY=.../resources/secrets.
+        It should find the file .../resources/secrets/secret1, containing "mySecret1".
+        """
+        os.environ['WDT_MODEL_SECRETS_NAME_DIR_PAIRS'] = "dirX=/noDir,dirY=" + self._resources_dir + "/secrets"
+        model = {'domainInfo': {'AdminUserName': '@@SECRET:dirY:secret1@@'}}
+        variables._clear_secret_token_map()
+        variables.substitute(model, {}, self.model_context)
+        self.assertEqual(model['domainInfo']['AdminUserName'], 'mySecret1')
+
+    def testSecretTokenNotFound(self):
+        try:
+            model = {'domainInfo': {'AdminUserName': '@@SECRET:noName:noKey@@'}}
+            variables.substitute(model, {}, self.model_context)
+        except VariableException:
+            pass
+        else:
+            self.fail('Test must raise VariableException when secret token is not found')
+
 
 if __name__ == '__main__':
     unittest.main()

core/src/test/resources/secrets/my-secrets/secret2

@@ -0,0 +1 @@
+mySecret2

core/src/test/resources/secrets/secret1

@@ -0,0 +1 @@
+mySecret1