ansible-test - Move code from _data to _util. (#75495)

* Update paths to match relocated files.
* Update ansible-test symlink paths.
* Update path classification.
* Update MANIFEST.in
* Update sanity test context paths.
* Update sanity ignores.
* Update shebang sanity test.
* Update configure-remoting-ps1 sanity test.
* Update BOTMETA.yml to reflect new paths.
* Update paths in collection loader comments.
* Update Makefile for ansible-test.
* Update docs.
* Add changelog fragment.

.github/BOTMETA.yml

@@ -121,16 +121,16 @@ files:
     notified: mattclay
   test/lib/:
     notified: mattclay
-  test/lib/ansible_test/_data/sanity/validate-modules:
+  test/lib/ansible_test/_util/controller/sanity/validate-modules:
     keywords:
     - validate-modules
     notified:
     - mattclay
-  test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/main.py:
+  test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/main.py:
     notified:
     - gundalow
     - sivel
-  test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py:
+  test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/schema.py:
     notified:
     - gundalow
     - sivel

MANIFEST.in

@@ -25,9 +25,10 @@ recursive-include packaging *
 recursive-include test/ansible_test *.py Makefile
 recursive-include test/integration *
 recursive-include test/lib/ansible_test/config *.yml *.template
-recursive-include test/lib/ansible_test/_data *.cfg *.ini *.json *.ps1 *.psd1 *.py *.sh *.txt *.yml coveragerc inventory
-recursive-include test/lib/ansible_test/_data/injector ansible ansible-config ansible-connection ansible-console ansible-doc ansible-galaxy ansible-playbook ansible-pull ansible-test ansible-vault pytest
-recursive-include test/lib/ansible_test/_data/sanity/validate-modules validate-modules
+recursive-include test/lib/ansible_test/_data *.cfg *.ini *.ps1 *.txt *.yml coveragerc inventory
+recursive-include test/lib/ansible_test/_util *.cfg *.json *.ps1 *.psd1 *.py *.sh *.txt *.yml
+recursive-include test/lib/ansible_test/_util/target/injector ansible ansible-config ansible-connection ansible-console ansible-doc ansible-galaxy ansible-inventory ansible-playbook ansible-pull ansible-test ansible-vault pytest
+recursive-include test/lib/ansible_test/_util/controller/sanity/validate-modules validate-modules
 recursive-include test/sanity *.json *.py *.txt
 recursive-include test/support *.py *.ps1 *.psm1 *.cs
 exclude test/sanity/code-smell/botmeta.*

bin/ansible-test

@@ -1 +1 @@
-../test/lib/ansible_test/_data/cli/ansible_test_cli_stub.py
+../test/lib/ansible_test/_util/controller/cli/ansible_test_cli_stub.py

changelogs/fragments/ansible-test-utils.yml

@@ -0,0 +1,2 @@
+minor_changes:
+  - ansible-test - Move code from ``_data`` directory to ``_util`` directory.

docs/docsite/rst/dev_guide/testing/sanity/bin-symlinks.rst

@@ -6,6 +6,6 @@ These files must reside in the ``lib/ansible/`` or ``test/lib/ansible_test/`` di
 
 This is required to allow ``ansible-test`` to work with containers and remote hosts when running from an installed version of Ansible.
 
-Symlinks for each entry point in ``bin/`` must also be present in ``test/lib/ansible_test/_data/injector/``.
+Symlinks for each entry point in ``bin/`` must also be present in ``test/lib/ansible_test/_util/target/injector/``.
 Each symlink should point to the ``python.py`` script in the same directory.
 This facilitates running with the correct Python interpreter and enabling code coverage.

docs/docsite/rst/dev_guide/testing/sanity/obsolete-files.rst

@@ -8,7 +8,7 @@ The new location (if any) is dependent on which directory has been made obsolete
 Below are some of the obsolete directories and their new locations:
 
 - All of ``test/runner/`` is now under ``test/lib/ansible_test/`` instead. The organization of files in the new directory has changed.
-- Most subdirectories of ``test/sanity/`` (with some exceptions) are now under ``test/lib/ansible_test/_data/sanity/`` instead.
+- Most subdirectories of ``test/sanity/`` (with some exceptions) are now under ``test/lib/ansible_test/_util/controller/sanity/`` instead.
 
 This error occurs most frequently for open pull requests which add or modify files in directories which are now obsolete.
 Make sure the branch you are working from is current so that changes can be made in the correct location.

docs/docsite/rst/dev_guide/testing_validate-modules.rst

@@ -52,7 +52,7 @@ Help
 Extending validate-modules
 ==========================
 
-The ``validate-modules`` tool has a `schema.py <https://github.com/ansible/ansible/blob/devel/test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py>`_ that is used to validate the YAML blocks, such as ``DOCUMENTATION`` and ``RETURNS``.
+The ``validate-modules`` tool has a `schema.py <https://github.com/ansible/ansible/blob/devel/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/schema.py>`_ that is used to validate the YAML blocks, such as ``DOCUMENTATION`` and ``RETURNS``.
 
 
 Codes

lib/ansible/utils/collection_loader/__init__.py

@@ -6,7 +6,7 @@
 #
 # 1) The controller implementation resides in the "lib/ansible/utils/collection_loader/" directory.
 #    It must function on all Python versions supported on the controller.
-# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_data/legacy_collection_loader/" directory.
+# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_util/target/legacy_collection_loader/" directory.
 #    It must function on all Python versions supported on managed hosts which are not supported by the controller.
 
 from __future__ import (absolute_import, division, print_function)

lib/ansible/utils/collection_loader/_collection_config.py

@@ -6,7 +6,7 @@
 #
 # 1) The controller implementation resides in the "lib/ansible/utils/collection_loader/" directory.
 #    It must function on all Python versions supported on the controller.
-# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_data/legacy_collection_loader/" directory.
+# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_util/target/legacy_collection_loader/" directory.
 #    It must function on all Python versions supported on managed hosts which are not supported by the controller.
 
 from __future__ import (absolute_import, division, print_function)

lib/ansible/utils/collection_loader/_collection_finder.py

@@ -6,7 +6,7 @@
 #
 # 1) The controller implementation resides in the "lib/ansible/utils/collection_loader/" directory.
 #    It must function on all Python versions supported on the controller.
-# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_data/legacy_collection_loader/" directory.
+# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_util/target/legacy_collection_loader/" directory.
 #    It must function on all Python versions supported on managed hosts which are not supported by the controller.
 
 from __future__ import (absolute_import, division, print_function)

lib/ansible/utils/collection_loader/_collection_meta.py

@@ -6,7 +6,7 @@
 #
 # 1) The controller implementation resides in the "lib/ansible/utils/collection_loader/" directory.
 #    It must function on all Python versions supported on the controller.
-# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_data/legacy_collection_loader/" directory.
+# 2) The ansible-test implementation resides in the "test/lib/ansible_test/_util/target/legacy_collection_loader/" directory.
 #    It must function on all Python versions supported on managed hosts which are not supported by the controller.
 
 from __future__ import (absolute_import, division, print_function)

test/ansible_test/Makefile

@@ -10,4 +10,4 @@ unit:
 
 .PHONY: validate-modules-unit
 validate-modules-unit:
-	PYTHONPATH=$(abspath ${CURDIR}/../lib/ansible_test/_data/sanity/validate-modules):$(abspath ${CURDIR}/../../lib) pytest validate-modules-unit ${FLAGS}
+	PYTHONPATH=$(abspath ${CURDIR}/../lib/ansible_test/_util/controller/sanity/validate-modules):$(abspath ${CURDIR}/../../lib) pytest validate-modules-unit ${FLAGS}

test/lib/ansible_test/_internal/ansible_util.py

@@ -24,6 +24,8 @@ from .util import (
     ANSIBLE_TEST_DATA_ROOT,
     ANSIBLE_BIN_PATH,
     ANSIBLE_SOURCE_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
+    ANSIBLE_TEST_TOOLS_ROOT,
     get_ansible_version,
 )
 
@@ -97,7 +99,7 @@ def ansible_environment(args, color=True, ansible_config=None):
         # ansible-connection only requires the injector for code coverage
         # the correct python interpreter is already selected using the sys.executable used to invoke ansible
         ansible.update(dict(
-            ANSIBLE_CONNECTION_PATH=os.path.join(ANSIBLE_TEST_DATA_ROOT, 'injector', 'ansible-connection'),
+            ANSIBLE_CONNECTION_PATH=os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'injector', 'ansible-connection'),
         ))
 
     if isinstance(args, PosixIntegrationConfig):
@@ -244,7 +246,7 @@ def check_pyyaml(args, version, required=True, quiet=False):
         pass
 
     python = find_python(version)
-    stdout, _dummy = run_command(args, [python, os.path.join(ANSIBLE_TEST_DATA_ROOT, 'yamlcheck.py')],
+    stdout, _dummy = run_command(args, [python, os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'yamlcheck.py')],
                                  capture=True, always=True)
 
     result = json.loads(stdout)
@@ -284,7 +286,7 @@ def get_collection_detail(args, python):  # type: (EnvironmentConfig, str) -> Co
     collection = data_context().content.collection
     directory = os.path.join(collection.root, collection.directory)
 
-    stdout = run_command(args, [python, os.path.join(ANSIBLE_TEST_DATA_ROOT, 'collection_detail.py'), directory], capture=True, always=True)[0]
+    stdout = run_command(args, [python, os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'collection_detail.py'), directory], capture=True, always=True)[0]
     result = json.loads(stdout)
     error = result.get('error')
 

test/lib/ansible_test/_internal/classification/__init__.py

@@ -804,30 +804,12 @@ class PathMapper:
                 'integration': 'ansible-test',  # run ansible-test self tests
             }
 
-        if path.startswith('test/lib/ansible_test/_data/sanity/'):
-            return {
-                'sanity': 'all',  # test infrastructure, run all sanity checks
-                'integration': 'ansible-test',  # run ansible-test self tests
-            }
-
         if path.startswith('test/lib/ansible_test/_internal/commands/units/'):
             return {
                 'units': 'all',  # test infrastructure, run all unit tests
                 'integration': 'ansible-test',  # run ansible-test self tests
             }
 
-        if path.startswith('test/lib/ansible_test/_data/units/'):
-            return {
-                'units': 'all',  # test infrastructure, run all unit tests
-                'integration': 'ansible-test',  # run ansible-test self tests
-            }
-
-        if path.startswith('test/lib/ansible_test/_data/pytest/'):
-            return {
-                'units': 'all',  # test infrastructure, run all unit tests
-                'integration': 'ansible-test',  # run ansible-test self tests
-            }
-
         if path.startswith('test/lib/ansible_test/_data/requirements/'):
             if name in (
                     'integration',
@@ -854,6 +836,18 @@ class PathMapper:
                         'integration': cloud_target,
                     }
 
+        if path.startswith('test/lib/ansible_test/_util/controller/sanity/') or path.startswith('test/lib/ansible_test/_util/target/sanity/'):
+            return {
+                'sanity': 'all',  # test infrastructure, run all sanity checks
+                'integration': 'ansible-test',  # run ansible-test self tests
+            }
+
+        if path.startswith('test/lib/ansible_test/_util/target/pytest/'):
+            return {
+                'units': 'all',  # test infrastructure, run all unit tests
+                'integration': 'ansible-test',  # run ansible-test self tests
+            }
+
         if path.startswith('test/lib/'):
             return all_tests(self.args)  # test infrastructure, run all tests
 

test/lib/ansible_test/_internal/classification/python.py

@@ -269,7 +269,8 @@ class ModuleUtilFinder(ast.NodeVisitor):
             path_map = (
                 ('^hacking/build_library/build_ansible/', 'build_ansible/'),
                 ('^lib/ansible/', 'ansible/'),
-                ('^test/lib/ansible_test/_data/sanity/validate-modules/', 'validate_modules/'),
+                ('^test/lib/ansible_test/_util/controller/sanity/validate-modules/', 'validate_modules/'),
+                ('^test/lib/ansible_test/_util/target/legacy_collection_loader/', 'legacy_collection_loader/'),
                 ('^test/units/', 'test/units/'),
                 ('^test/lib/ansible_test/_internal/', 'ansible_test/_internal/'),
                 ('^test/integration/targets/.*/ansible_collections/(?P<ns>[^/]*)/(?P<col>[^/]*)/', r'ansible_collections/\g<ns>/\g<col>/'),

test/lib/ansible_test/_internal/commands/coverage/combine.py

@@ -17,7 +17,7 @@ from ...io import (
 )
 
 from ...util import (
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TOOLS_ROOT,
     display,
     ApplicationError,
 )
@@ -181,7 +181,7 @@ def _command_coverage_combine_powershell(args):
     coverage_files = get_powershell_coverage_files()
 
     def _default_stub_value(source_paths):
-        cmd = ['pwsh', os.path.join(ANSIBLE_TEST_DATA_ROOT, 'coverage_stub.ps1')]
+        cmd = ['pwsh', os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'coverage_stub.ps1')]
         cmd.extend(source_paths)
 
         stubs = json.loads(run_command(args, cmd, capture=True, always=True)[0])

test/lib/ansible_test/_internal/commands/integration/__init__.py

@@ -72,7 +72,7 @@ from ...util import (
     remove_tree,
     find_executable,
     raw_command,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TOOLS_ROOT,
     SUPPORTED_PYTHON_VERSIONS,
     get_hash,
 )
@@ -1132,7 +1132,7 @@ class EnvironmentDescription:
         versions += SUPPORTED_PYTHON_VERSIONS
         versions += list(set(v.split('.', 1)[0] for v in SUPPORTED_PYTHON_VERSIONS))
 
-        version_check = os.path.join(ANSIBLE_TEST_DATA_ROOT, 'versions.py')
+        version_check = os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'versions.py')
         python_paths = dict((v, find_executable('python%s' % v, required=False)) for v in sorted(versions))
         pip_paths = dict((v, find_executable('pip%s' % v, required=False)) for v in sorted(versions))
         program_versions = dict((v, self.get_version([python_paths[v], version_check], warnings)) for v in sorted(python_paths) if python_paths[v])

test/lib/ansible_test/_internal/commands/sanity/__init__.py

@@ -22,7 +22,8 @@ from ...util import (
     load_plugins,
     parse_to_list_of_dict,
     ABC,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_CONTROLLER_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
     is_binary_file,
     read_lines_without_comments,
     get_available_python_versions,
@@ -80,7 +81,8 @@ from ...content_config import (
 )
 
 COMMAND = 'sanity'
-SANITY_ROOT = os.path.join(ANSIBLE_TEST_DATA_ROOT, 'sanity')
+SANITY_ROOT = os.path.join(ANSIBLE_TEST_CONTROLLER_ROOT, 'sanity')
+TARGET_SANITY_ROOT = os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'sanity')
 
 
 def command_sanity(args):

test/lib/ansible_test/_internal/commands/sanity/bin_symlinks.py

@@ -28,7 +28,7 @@ from ...payload import (
 
 from ...util import (
     ANSIBLE_BIN_PATH,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
 )
 
 
@@ -57,7 +57,7 @@ class BinSymlinksTest(SanityVersionNeutral):
         bin_names = os.listdir(bin_root)
         bin_paths = sorted(os.path.join(bin_root, path) for path in bin_names)
 
-        injector_root = os.path.join(ANSIBLE_TEST_DATA_ROOT, 'injector')
+        injector_root = os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'injector')
         injector_names = os.listdir(injector_root)
 
         errors = []  # type: t.List[t.Tuple[str, str]]

test/lib/ansible_test/_internal/commands/sanity/compile.py

@@ -12,7 +12,7 @@ from . import (
     SanityFailure,
     SanitySuccess,
     SanityTargets,
-    SANITY_ROOT,
+    TARGET_SANITY_ROOT,
 )
 
 from ...target import (
@@ -53,7 +53,7 @@ class CompileTest(SanityMultipleVersion):
 
         paths = [target.path for target in targets.include]
 
-        cmd = [find_python(python_version), os.path.join(SANITY_ROOT, 'compile', 'compile.py')]
+        cmd = [find_python(python_version), os.path.join(TARGET_SANITY_ROOT, 'compile', 'compile.py')]
 
         data = '\n'.join(paths)
 

test/lib/ansible_test/_internal/commands/sanity/import.py

@@ -12,7 +12,7 @@ from . import (
     SanityFailure,
     SanitySuccess,
     SanitySkipped,
-    SANITY_ROOT,
+    TARGET_SANITY_ROOT,
 )
 
 from ...target import (
@@ -138,8 +138,8 @@ class ImportTest(SanityMultipleVersion):
             importer_path = os.path.join(virtual_environment_bin, 'importer.py')
             yaml_to_json_path = os.path.join(virtual_environment_bin, 'yaml_to_json.py')
             if not args.explain:
-                os.symlink(os.path.abspath(os.path.join(SANITY_ROOT, 'import', 'importer.py')), importer_path)
-                os.symlink(os.path.abspath(os.path.join(SANITY_ROOT, 'import', 'yaml_to_json.py')), yaml_to_json_path)
+                os.symlink(os.path.abspath(os.path.join(TARGET_SANITY_ROOT, 'import', 'importer.py')), importer_path)
+                os.symlink(os.path.abspath(os.path.join(TARGET_SANITY_ROOT, 'import', 'yaml_to_json.py')), yaml_to_json_path)
 
             # activate the virtual environment
             env['PATH'] = '%s:%s' % (virtual_environment_bin, env['PATH'])

test/lib/ansible_test/_internal/commands/sanity/pylint.py

@@ -136,10 +136,11 @@ class PylintTest(SanitySingleVersion):
         if data_context().content.collection:
             add_context(remaining_paths, 'collection', lambda p: True)
         else:
-            add_context(remaining_paths, 'validate-modules', filter_path('test/lib/ansible_test/_data/sanity/validate-modules/'))
+            add_context(remaining_paths, 'validate-modules', filter_path('test/lib/ansible_test/_util/controller/sanity/validate-modules/'))
             add_context(remaining_paths, 'validate-modules-unit', filter_path('test/lib/ansible_test/tests/validate-modules-unit/'))
-            add_context(remaining_paths, 'sanity', filter_path('test/lib/ansible_test/_data/sanity/'))
-            add_context(remaining_paths, 'legacy-collection-loader', filter_path('test/lib/ansible_test/_data/legacy_collection_loader/'))
+            add_context(remaining_paths, 'sanity', filter_path('test/lib/ansible_test/_util/controller/sanity/'))
+            add_context(remaining_paths, 'sanity', filter_path('test/lib/ansible_test/_util/target/sanity/'))
+            add_context(remaining_paths, 'legacy-collection-loader', filter_path('test/lib/ansible_test/_util/target/legacy_collection_loader/'))
             add_context(remaining_paths, 'ansible-test', filter_path('test/lib/'))
             add_context(remaining_paths, 'test', filter_path('test/'))
             add_context(remaining_paths, 'hacking', filter_path('hacking/'))

test/lib/ansible_test/_internal/commands/units/__init__.py

@@ -21,6 +21,7 @@ from ...util import (
     CONTROLLER_PYTHON_VERSIONS,
     REMOTE_ONLY_PYTHON_VERSIONS,
     ANSIBLE_LIB_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
 )
 
 from ...util_common import (
@@ -203,7 +204,7 @@ def command_units(args):
             plugins.append('ansible_pytest_collections')
 
         if plugins:
-            env['PYTHONPATH'] += ':%s' % os.path.join(ANSIBLE_TEST_DATA_ROOT, 'pytest/plugins')
+            env['PYTHONPATH'] += ':%s' % os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'pytest/plugins')
             env['PYTEST_PLUGINS'] = ','.join(plugins)
 
         if args.collect_only:
@@ -262,7 +263,7 @@ def get_units_ansible_python_path(args, test_context):  # type: (UnitsConfig, st
         # legacy collection loader required by all python versions not supported by the controller
         write_text_file(os.path.join(ansible_test_path, '__init__.py'), '', True)
         write_text_file(os.path.join(ansible_test_path, '_internal', '__init__.py'), '', True)
-        os.symlink(os.path.join(ANSIBLE_TEST_DATA_ROOT, 'legacy_collection_loader'), os.path.join(ansible_test_path, '_internal', 'legacy_collection_loader'))
+        os.symlink(os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'legacy_collection_loader'), os.path.join(ansible_test_path, '_internal', 'legacy_collection_loader'))
     elif test_context == TestContext.modules:
         # only non-collection ansible module tests should have access to ansible built-in modules
         os.symlink(os.path.join(ANSIBLE_LIB_ROOT, 'modules'), os.path.join(ansible_path, 'modules'))

test/lib/ansible_test/_internal/core_ci.py

@@ -28,7 +28,7 @@ from .io import (
 from .util import (
     ApplicationError,
     display,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
 )
 
 from .util_common import (
@@ -325,7 +325,7 @@ class AnsibleCoreCI:
         display.info('Initializing new %s/%s instance %s.' % (self.platform, self.version, self.instance_id), verbosity=1)
 
         if self.platform == 'windows':
-            winrm_config = read_text_file(os.path.join(ANSIBLE_TEST_DATA_ROOT, 'setup', 'ConfigureRemotingForAnsible.ps1'))
+            winrm_config = read_text_file(os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'setup', 'ConfigureRemotingForAnsible.ps1'))
         else:
             winrm_config = None
 

test/lib/ansible_test/_internal/delegation.py

@@ -49,7 +49,7 @@ from .util import (
     common_environment,
     display,
     ANSIBLE_BIN_PATH,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
     ANSIBLE_LIB_ROOT,
     ANSIBLE_TEST_ROOT,
     tempdir,
@@ -297,7 +297,7 @@ def delegate_docker(args, exclude, require):
 
             test_id = docker_run(args, test_image, options=test_options)
 
-            setup_sh = read_text_file(os.path.join(ANSIBLE_TEST_DATA_ROOT, 'setup', 'docker.sh'))
+            setup_sh = read_text_file(os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'setup', 'docker.sh'))
 
             ssh_keys_sh = get_ssh_key_setup(SshKey(args))
 

test/lib/ansible_test/_internal/executor.py

@@ -26,6 +26,7 @@ from .util import (
     find_python,
     cmd_quote,
     ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TOOLS_ROOT,
     str_to_version,
     version_to_str,
 )
@@ -99,7 +100,7 @@ def get_openssl_version(args, python, python_version):  # type: (EnvironmentConf
     if not python_version.startswith('2.'):
         # OpenSSL version checking only works on Python 3.x.
         # This should be the most accurate, since it is the Python we will be using.
-        version = json.loads(run_command(args, [python, os.path.join(ANSIBLE_TEST_DATA_ROOT, 'sslcheck.py')], capture=True, always=True)[0])['version']
+        version = json.loads(run_command(args, [python, os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'sslcheck.py')], capture=True, always=True)[0])['version']
 
         if version:
             display.info('Detected OpenSSL version %s under Python %s.' % (version_to_str(version), python_version), verbosity=1)

test/lib/ansible_test/_internal/manage_ci.py

@@ -19,7 +19,7 @@ from .util import (
     Display,
     cmd_quote,
     display,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
 )
 
 from .util_common import (
@@ -276,7 +276,7 @@ class ManagePosixCI:
         """Configure remote host for testing.
         :type python_version: str
         """
-        template = ShellScriptTemplate(read_text_file(os.path.join(ANSIBLE_TEST_DATA_ROOT, 'setup', 'remote.sh')))
+        template = ShellScriptTemplate(read_text_file(os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'setup', 'remote.sh')))
         setup_sh = template.substitute(
             platform=self.core_ci.platform,
             platform_version=self.core_ci.version,
@@ -401,7 +401,7 @@ class ManagePosixCI:
 
 def get_ssh_key_setup(ssh_key):  # type: (SshKey) -> str
     """Generate and return a script to configure SSH keys on a host."""
-    template = ShellScriptTemplate(read_text_file(os.path.join(ANSIBLE_TEST_DATA_ROOT, 'setup', 'ssh-keys.sh')))
+    template = ShellScriptTemplate(read_text_file(os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'setup', 'ssh-keys.sh')))
 
     ssh_keys_sh = template.substitute(
         ssh_public_key=ssh_key.pub_contents,

test/lib/ansible_test/_internal/payload.py

@@ -47,7 +47,7 @@ ANSIBLE_BIN_SYMLINK_MAP = {
     'ansible-inventory': 'ansible',
     'ansible-playbook': 'ansible',
     'ansible-pull': 'ansible',
-    'ansible-test': '../test/lib/ansible_test/_data/cli/ansible_test_cli_stub.py',
+    'ansible-test': '../test/lib/ansible_test/_util/controller/cli/ansible_test_cli_stub.py',
     'ansible-vault': 'ansible',
 }
 

test/lib/ansible_test/_internal/util.py

@@ -97,8 +97,14 @@ if not os.path.exists(ANSIBLE_LIB_ROOT):
     ANSIBLE_SOURCE_ROOT = ANSIBLE_ROOT
 
 ANSIBLE_TEST_DATA_ROOT = os.path.join(ANSIBLE_TEST_ROOT, '_data')
+ANSIBLE_TEST_UTIL_ROOT = os.path.join(ANSIBLE_TEST_ROOT, '_util')
 ANSIBLE_TEST_CONFIG_ROOT = os.path.join(ANSIBLE_TEST_ROOT, 'config')
 
+ANSIBLE_TEST_CONTROLLER_ROOT = os.path.join(ANSIBLE_TEST_UTIL_ROOT, 'controller')
+ANSIBLE_TEST_TARGET_ROOT = os.path.join(ANSIBLE_TEST_UTIL_ROOT, 'target')
+
+ANSIBLE_TEST_TOOLS_ROOT = os.path.join(ANSIBLE_TEST_CONTROLLER_ROOT, 'tools')
+
 # Modes are set to allow all users the same level of access.
 # This permits files to be used in tests that change users.
 # The only exception is write access to directories for the user creating them.
@@ -265,7 +271,7 @@ def generate_pip_command(python):
     :type python: str
     :rtype: list[str]
     """
-    return [python, os.path.join(ANSIBLE_TEST_DATA_ROOT, 'quiet_pip.py')]
+    return [python, os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'quiet_pip.py')]
 
 
 def raw_command(cmd, capture=False, env=None, data=None, cwd=None, explain=False, stdin=None, stdout=None,

test/lib/ansible_test/_internal/util_common.py

@@ -29,6 +29,7 @@ from .util import (
     raw_command,
     read_lines_without_comments,
     ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TARGET_ROOT,
     ApplicationError,
     cmd_quote,
     SubprocessError,
@@ -416,7 +417,7 @@ def intercept_command(args, cmd, target_name, env, capture=False, data=None, cwd
     cmd = list(cmd)
     version = python_version or args.python_version
     interpreter = virtualenv or find_python(version)
-    inject_path = os.path.join(ANSIBLE_TEST_DATA_ROOT, 'injector')
+    inject_path = os.path.join(ANSIBLE_TEST_TARGET_ROOT, 'injector')
 
     if not virtualenv:
         # injection of python into the path is required when not activating a virtualenv

test/lib/ansible_test/_internal/venv.py

@@ -16,7 +16,7 @@ from .util import (
     find_python,
     SubprocessError,
     get_available_python_versions,
-    ANSIBLE_TEST_DATA_ROOT,
+    ANSIBLE_TEST_TOOLS_ROOT,
     display,
     remove_tree,
 )
@@ -128,7 +128,7 @@ def get_python_real_prefix(args, path):  # type: (EnvironmentConfig, str) -> t.O
     """
     Return the real prefix of the specified interpreter or None if the interpreter is not a virtual environment created by 'virtualenv'.
     """
-    cmd = [path, os.path.join(os.path.join(ANSIBLE_TEST_DATA_ROOT, 'virtualenvcheck.py'))]
+    cmd = [path, os.path.join(os.path.join(ANSIBLE_TEST_TOOLS_ROOT, 'virtualenvcheck.py'))]
     check_result = json.loads(run_command(args, cmd, capture=True, always=True)[0])
     real_prefix = check_result['real_prefix']
     return real_prefix

test/lib/ansible_test/_util/controller/sanity/code-smell/runtime-metadata.py

@@ -107,8 +107,7 @@ def get_collection_version():
     """Return current collection version, or None if it is not available"""
     import importlib.util
 
-    collection_detail_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))),
-                                          'collection_detail.py')
+    collection_detail_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'tools', 'collection_detail.py')
     collection_detail_spec = importlib.util.spec_from_file_location('collection_detail', collection_detail_path)
     collection_detail = importlib.util.module_from_spec(collection_detail_spec)
     sys.modules['collection_detail'] = collection_detail

test/lib/ansible_test/_util/controller/sanity/code-smell/shebang.py

@@ -71,6 +71,8 @@ def main():
                 is_module = True
             elif path.startswith('test/lib/ansible_test/_data/'):
                 pass
+            elif path.startswith('test/lib/ansible_test/_util/'):
+                pass
             elif path.startswith('lib/') or path.startswith('test/lib/'):
                 if executable:
                     print('%s:%d:%d: should not be executable' % (path, 0, 0))