misc changelog and docstring cleanup (#85148)

* deferred listify_lookup_plugin_terms deprecation

Co-authored-by: Matt Clay <matt@mystile.com>

changelogs/fragments/templates_types_datatagging.yml

@@ -1,21 +1,19 @@
-# DTFIX-RELEASE: document EncryptedString replacing AnsibleVaultEncryptedUnicode
-
 major_changes:
   - variables - The type system underlying Ansible's variable storage has been significantly overhauled and formalized.
-    Attempts to store unsupported Python object types in variables will now result in an error.  # DTFIX-RELEASE: link to type system docs TBD
+    Attempts to store unsupported Python object types in variables now more consistently yields early warnings or errors.
   - variables - To support new Ansible features, many variable objects are now represented by subclasses of their respective native Python types.
     In most cases, they behave indistinguishably from their original types, but some Python libraries do not handle builtin object subclasses properly.
-    Custom plugins that interact with such libraries may require changes to convert and pass the native types.  # DTFIX-RELEASE: link to plugin/data tagging API docs TBD
+    Custom plugins that interact with such libraries may require changes to convert and pass the native types.
   - ansible-test - Packages beneath ``module_utils`` can now contain ``__init__.py`` files.
   - Jinja plugins - Jinja builtin filter and test plugins are now accessible via their fully-qualified names ``ansible.builtin.{name}``.
 
 minor_changes:
   - templating - Templating errors now provide more information about both the location and context of the error, especially for deeply-nested and/or indirected templating scenarios.
   - templating - Handling of omitted values is now a first-class feature of the template engine, and is usable in all Ansible Jinja template contexts.
-    Any template that resolves to ``omit`` is automatically removed from its parent container during templating.  # DTFIX-RELEASE: porting guide entry
+    Any template that resolves to ``omit`` is automatically removed from its parent container during templating.
   - templating - Unified ``omit`` behavior now requires that plugins calling ``Templar.template()`` handle cases where the entire template result is omitted,
     by catching the ``AnsibleValueOmittedError`` that is raised.
-    Previously, this condition caused a randomly-generated string marker to appear in the template result.  # DTFIX-RELEASE: porting guide entry?
+    Previously, this condition caused a randomly-generated string marker to appear in the template result.
   - templating - Template evaluation is lazier than in previous versions.
     Template expressions which resolve only portions of a data structure no longer result in the entire structure being templated.
   - handlers - Templated handler names with syntax errors, or that resolve to ``omit`` are now skipped like handlers with undefined variables in their name.
@@ -31,7 +29,7 @@ minor_changes:
     This supersedes the use of the ``AnsibleError`` arg ``orig_exc`` to represent the cause.
     Specifying ``orig_exc`` as the cause is still permitted.
     Failure to use ``raise ... from`` when ``orig_exc`` is set will result in a warning.
-    Additionally, if the two cause exceptions do not match, a warning will be issued.  # DTFIX-RELEASE: this needs a porting guide entry
+    Additionally, if the two cause exceptions do not match, a warning will be issued.
   - ansible-test - The ``yamllint`` sanity test now enforces string values for the ``!vault`` tag.
   - warnings - All warnings (including deprecation warnings) issued during a task's execution are now accessible via the ``warnings`` and ``deprecations`` keys on the task result.
   - troubleshooting - Tracebacks can be collected and displayed for most errors, warnings, and deprecation warnings (including those generated by modules).
@@ -53,7 +51,7 @@ breaking_changes:
     Previously, ``omit`` placeholders could remain embedded in loop items after templating and be used as an ``omit`` for task templating.
     Now, values resolving to ``omit`` are dropped immediately when loop items are templated.
     To turn missing values into an ``omit`` for task templating, use ``| default(omit)``.
-    This solution is backwards compatible with previous versions of ansible-core.  # DTFIX-RELEASE: porting guide entry with examples
+    This solution is backward-compatible with previous versions of ansible-core.
   - serialization of ``omit`` sentinel - Serialization of variables containing ``omit`` sentinels (e.g., by the ``to_json`` and ``to_yaml`` filters or ``ansible-inventory``) will fail if the variable has not completed templating.
     Previously, serialization succeeded with placeholder strings emitted in the serialized output.
   - conditionals - Conditional expressions that result in non-boolean values are now an error by default.
@@ -88,9 +86,8 @@ breaking_changes:
     This exception must be handled to allow the plugin to return a ``Marker``, or the plugin must opt-in to accepting ``Marker`` values.  # DTFIX-RELEASE: mention the decorator
   - templating - Many Jinja plugins (filters, lookups, tests) and methods previously silently ignored undefined inputs, which often masked subtle errors.
     Passing an undefined argument to a Jinja plugin or method that does not declare undefined support now results in an undefined value.  # DTFIX-RELEASE: common examples, porting guide, `is defined`, `is undefined`, etc; porting guide should also mention that overly-broad exception handling may mask Undefined errors; also that lazy handling of Undefined can invoke a plugin and bomb out in the middle where it was previously never invoked (plugins with side effects, just don't)
-  - lookup plugins - Lookup plugins called as `with_(lookup)` will no longer have the `_subdir` attribute set. # DTFIX-RELEASE: porting guide re: `ansible_lookup_context`
+  - lookup plugins - Lookup plugins called as `with_(lookup)` will no longer have the `_subdir` attribute set.
   - lookup plugins - ``terms`` will always be passed to ``run`` as the first positional arg, where previously it was sometimes passed as a keyword arg when using ``with_`` syntax.
-  - callback plugins - The structure of the ``exception``, ``warnings`` and ``deprecations`` values visible to callbacks has changed. Callbacks that inspect or serialize these values may require special handling. # DTFIX-RELEASE: porting guide re ErrorDetail/WarningMessageDetail/DeprecationMessageDetail
   - modules - Ansible modules using ``sys.excepthook`` must use a standard ``try/except`` instead.
   - templating - Access to ``_`` prefixed attributes and methods, and methods with known side effects, is no longer permitted.
     In cases where a matching mapping key is present, the associated value will be returned instead of an error. 
@@ -108,17 +105,9 @@ security_fixes:
     Plugins which have not been updated to preserve trust while manipulating strings may inadvertently cause them to lose their trusted status.
   - templating - Changes to conditional expression handling removed numerous instances of insecure multi-pass templating (which could result in execution of untrusted template expressions).
 
-known_issues:
-  - variables - The values ``None``, ``True`` and ``False`` cannot be tagged because they are singletons. Attempts to apply tags to these values will be silently ignored.
-  - variables - Tagged values cannot be used for dictionary keys in many circumstances. # DTFIX-RELEASE: Explain this in more detail.
-  - templating - Any string value starting with ``#jinja2:`` which is templated will always be interpreted as Jinja2 configuration overrides.
-                 To include this literal value at the start of a string, a space or other character must precede it.
-
 bugfixes:
   - module defaults - Module defaults are no longer templated unless they are used by a task that does not override them.
     Previously, all module defaults for all modules were templated for every task.
-  - omitting task args - Use of omit for task args now properly falls back to args of lower precedence, such as module defaults.
-    Previously an omitted value would obliterate values of lower precedence.  # DTFIX-RELEASE: do we need obliterate, is this a breaking change?
   - regex_search filter - Corrected return value documentation to reflect None (not empty string) for no match.
   - first_found lookup - Corrected return value documentation to reflect None (not empty string) for no files found.
   - vars lookup - The ``default`` substitution only applies when trying to look up a variable which is not defined.
@@ -153,13 +142,12 @@ deprecated_features:
   - templating - The ``disable_lookups`` option has no effect, since plugins must be updated to apply trust before any templating can be performed.
   - to_yaml/to_nice_yaml filters - Implicit YAML dumping of vaulted value ciphertext is deprecated.
     Set `dump_vault_tags` to explicitly specify the desired behavior.
-  - plugins - The ``listify_lookup_plugin_terms`` function is obsolete and in most cases no longer needed.  # DTFIX-RELEASE: add a porting guide entry for this
   - plugin error handling - The ``AnsibleError`` constructor arg ``suppress_extended_error`` is deprecated.
     Using ``suppress_extended_error=True`` has the same effect as ``show_content=False``.
   - config - The ``ACTION_WARNINGS`` config has no effect. It previously disabled command warnings, which have since been removed.
   - templating - Support for enabling Jinja2 extensions (not plugins) has been deprecated.
   - playbook variables - The ``play_hosts`` variable has been deprecated, use ``ansible_play_batch`` instead.
-  - bool filter - Support for coercing unrecognized input values (including None) has been deprecated. Consult the filter documentation for acceptable values, or consider use of the ``truthy`` and ``falsy`` tests.  # DTFIX-RELEASE: porting guide
+  - bool filter - Support for coercing unrecognized input values (including None) has been deprecated. Consult the filter documentation for acceptable values, or consider use of the ``truthy`` and ``falsy`` tests.
   - oneline callback - The ``oneline`` callback and its associated ad-hoc CLI args (``-o``, ``--one-line``) are deprecated.
   - tree callback - The ``tree`` callback and its associated ad-hoc CLI args (``-t``, ``--tree``) are deprecated.
   - CLI - The ``--inventory-file`` option alias is deprecated. Use the ``-i`` or ``--inventory`` option instead.

lib/ansible/_internal/_templating/_jinja_bits.py

@@ -71,6 +71,11 @@ from ansible.vars.hostvars import HostVars, HostVarsVars
 from ...module_utils.datatag import native_type_name
 
 JINJA2_OVERRIDE = '#jinja2:'
+"""
+String values prefixed with this sequence are interpreted as templates, even without template delimiters.
+The values following this prefix up to the first newline are parsed as Jinja2 template overrides.
+To include this literal value at the start of a string, a space or other character must precede it.
+"""
 
 display = Display()
 

lib/ansible/module_utils/_internal/_datatag/__init__.py

@@ -37,6 +37,12 @@ _ANSIBLE_TAGGED_OBJECT_SLOTS = tuple(('_ansible_tags_mapping',))
 # shared empty frozenset for default values
 _empty_frozenset: t.FrozenSet = frozenset()
 
+# Technical Notes
+#
+# Tagged values compare (and thus hash) the same as their base types, so a value that differs only by its tags will appear identical to non-tag-aware code.
+# This will affect storage and update of tagged values in dictionary keys, sets, etc. While tagged values can be used as keys in hashable collections,
+# updating a key usually requires removal and re-addition.
+
 
 class AnsibleTagHelper:
     """Utility methods for working with Ansible data tags."""
@@ -914,10 +920,12 @@ class _AnsibleTaggedTuple(tuple, AnsibleTaggedObject):
         return super()._copy_collection()
 
 
-# This set gets augmented with additional types when some controller-only types are imported.
-# While we could proxy or subclass builtin singletons, they're idiomatically compared with "is" reference
-# equality, which we can't customize.
 _untaggable_types = {type(None), bool}
+"""
+Attempts to apply tags to values of these types will be silently ignored.
+While we could proxy or subclass builtin singletons, they're idiomatically compared with "is" reference equality, which we can't customize.
+This set gets augmented with additional types when some controller-only types are imported.
+"""
 
 # noinspection PyProtectedMember
 _ANSIBLE_ALLOWED_VAR_TYPES = frozenset({type(None), bool}) | set(AnsibleTaggedObject._tagged_type_map) | set(AnsibleTaggedObject._tagged_type_map.values())

lib/ansible/module_utils/_internal/_json/_profiles/__init__.py

@@ -203,6 +203,8 @@ class _JSONSerializationProfile(t.Generic[_T_encoder, _T_decoder]):
 
     @classmethod
     def handle_key(cls, k: t.Any) -> t.Any:
+        # NOTE: Since JSON requires string keys, there is no support for preserving tags on dictionary keys during serialization.
+
         if not isinstance(k, str):  # DTFIX-FUTURE: optimize this to use all known str-derived types in type map / allowed types
             raise TypeError(f'Key of type {type(k).__name__!r} is not JSON serializable by the {cls.profile_name!r} profile.')
 

lib/ansible/utils/listify.py

@@ -26,10 +26,12 @@ __all__ = ['listify_lookup_plugin_terms']
 
 
 def listify_lookup_plugin_terms(terms, templar=None, fail_on_undefined=True):
-    display.deprecated(
-        msg='"listify_lookup_plugin_terms" is obsolete and in most cases unnecessary',
-        version='2.23',
-    )
+    # deprecated: description="Calling listify_lookup_plugin_terms function is not necessary; the function should be deprecated." core_version="2.23"
+    # display.deprecated(
+    #     msg='The "listify_lookup_plugin_terms" function is not required for lookup terms to be templated.',
+    #     version='2.27',
+    #     help_text='If needed, implement custom `strip` or list-wrapping in the caller.',
+    # )
 
     if isinstance(terms, str):
         terms = terms.strip()

test/units/utils/test_listify.py

@@ -18,11 +18,11 @@ from ansible.utils.listify import listify_lookup_plugin_terms
     (["foo"], ["foo"]),
 ), ids=str)
 def test_listify_lookup_plugin_terms(test_input: t.Any, expected: t.Any, mocker: pytest_mock.MockerFixture) -> None:
-    deprecated = mocker.patch.object(Display(), 'deprecated')
+    with mocker.patch.object(Display(), 'deprecated'):  # as deprecated:
+        assert listify_lookup_plugin_terms(test_input) == expected
 
-    assert listify_lookup_plugin_terms(test_input) == expected
-
-    deprecated.assert_called_once_with(
-        msg='"listify_lookup_plugin_terms" is obsolete and in most cases unnecessary',
-        version='2.23',
-    )
+    # deprecated: description="Calling listify_lookup_plugin_terms function is not necessary; the function should be deprecated." core_version="2.23"
+    # deprecated.assert_called_once_with(
+    #     msg='"listify_lookup_plugin_terms" is obsolete and in most cases unnecessary',
+    #     version='2.23',
+    # )