dict_kwargs restricted to classes with an unresolved **kwargs

CHANGELOG.rst

@@ -104,6 +104,14 @@ Changed
 - Relative paths in config files are now resolved without changing the process
   working directory (`#979
   <https://github.com/mauvilsa/jsonargparse/pull/979>`__).
+- ``dict_kwargs`` in a subclass spec is now restricted to classes that have a
+  ``**kwargs`` the parameter resolvers are unable to resolve, which is noted in
+  the class help. For other classes, a ``dict_kwargs`` key that is not a
+  parameter of the class fails, instead of being silently ignored or only
+  noticed as a ``TypeError`` when the class is instantiated. Resolved parameters
+  are meant to be given in ``init_args``, and the ``jsonschema`` completion type
+  only describes them there, see :ref:`unresolved-parameters` (`#981
+  <https://github.com/mauvilsa/jsonargparse/pull/981>`__).
 
 Removed
 ^^^^^^^

DOCUMENTATION.rst

@@ -1612,7 +1612,9 @@ A wide range of type hints is supported for signature parameters, see
   unless they are required.
 
 - The ``skip`` parameter excludes arguments, e.g.
-  ``parser.add_method_arguments(MyClass, 'mymethod', skip={'baz'})``.
+  ``parser.add_method_arguments(MyClass, 'mymethod', skip={'baz'})``. In a
+  subclass spec, a skipped parameter can still be given in ``dict_kwargs``, see
+  :ref:`unresolved-parameters`.
 
 .. note::
 
@@ -1772,6 +1774,8 @@ assumptions resolver, based on assumptions about class inheritance, is the
 fallback for when AST fails. The stubs resolver, which uses ``*.pyi`` stub
 files, is applied on top of both.
 
+.. _unresolved-parameters:
+
 Unresolved parameters
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -1795,8 +1799,7 @@ Take for example the following parsing and instantiation:
 
     class MyClass:
         def __init__(self, foo: int = 0, **kwargs):
-            super().__init__(**kwargs)
-            ...
+            self.kwargs = kwargs
 
 
     MyClass.__module__ = "jsonargparse_tests"
@@ -1811,8 +1814,8 @@ Take for example the following parsing and instantiation:
     cfg = parser.parse_args()
     cfg_init = parser.instantiate(cfg)
 
-If ``MyClass.__init__`` has ``**kwargs`` with some unresolved parameters, the
-following could be a valid config file:
+Since the resolvers can't determine where the ``**kwargs`` of
+``MyClass.__init__`` go, the following is a valid config file:
 
 .. code-block:: yaml
 
@@ -1823,7 +1826,18 @@ following could be a valid config file:
       bar: 2
 
 The value for ``bar`` is not validated, but the class is instantiated as
-``MyClass(foo=1, bar=2)``.
+``MyClass(foo=1, bar=2)``. The help of a class, e.g. ``--myclass.help=MyClass``,
+notes when it accepts extra keyword arguments through ``dict_kwargs``.
+
+Resolved parameters are meant to be given in ``init_args``. When a class has no
+unresolved ``**kwargs``, a ``dict_kwargs`` key that is not one of its parameters
+fails during parsing. Keys that the class does accept are moved to
+``init_args``, so that configs keep working when an improvement of the resolvers
+turns an unresolved parameter into a resolved one.
+
+A parameter excluded with ``skip`` is still a parameter of the class, so
+``dict_kwargs`` accepts it. This is how to give a value to a parameter that had
+to be skipped, e.g. an untyped mandatory one.
 
 Assumptions resolver
 ^^^^^^^^^^^^^^^^^^^^
@@ -3358,12 +3372,15 @@ Subclasses and types that are used in more than one place are added once to
 The schema is meant to accept what the parser accepts, but for subclass types it
 is stricter. A string is accepted, since it can be a class path or a path to a
 sub-config file. An object is only accepted for the known subclasses, i.e. one
-with ``class_path``, ``init_args`` (required only for the subclasses that have a
-required init parameter) and ``dict_kwargs``. Accepting any ``class_path`` would
-keep tools from suggesting the known subclasses and from pointing out a class
-path that has a typo or is not the accepted import path, and its ``init_args``
-would go undescribed. Any ``class_path`` is accepted only when a type has no
-known subclass, and then its ``init_args`` are not described.
+with ``class_path`` and ``init_args`` (required only for the subclasses that
+have a required init parameter). Accepting any ``class_path`` would keep tools
+from suggesting the known subclasses and from pointing out a class path that has
+a typo or is not the accepted import path, and its ``init_args`` would go
+undescribed. Any ``class_path`` is accepted only when a type has no known
+subclass, and then its ``init_args`` are not described. Likewise,
+``dict_kwargs`` is only accepted for subclasses that have an unresolved
+``**kwargs``, so a skipped parameter given there is rejected even though the
+parser accepts it, see :ref:`unresolved-parameters`.
 
 A union that has a subtype accepting anything, i.e. ``Any`` or an unvalidated
 type, is kept as ``{"anyOf": [..., {}]}`` instead of the equivalent ``{}``, so

jsonargparse/_actions.py

@@ -418,6 +418,8 @@ def print_help(self, call_args):
         if partial_skip_args:
             sub_add_kwargs.setdefault("skip", set()).update(partial_skip_args)
         subparser.add_class_arguments(val_class, dest, **sub_add_kwargs)
+        if subparser._accepted_kwargs[dest] is True:
+            subparser.epilog = "Extra keyword arguments are accepted through dict_kwargs."
         subparser._inner_parser = True
         remove_actions(subparser, (_HelpAction, _ActionPrintConfig, _ActionConfigLoad))
         args = self.get_args_after_opt(parser.args)

jsonargparse/_completions_jsonschema.py

@@ -544,14 +544,12 @@ def subclass_def(self, class_type, action) -> dict:
 
     def class_path_schema(self, class_path: str, action) -> dict:
         schema = new_object(get_doc_short_description(import_object(class_path)))
-        init_args_schema = self.class_parser_schema(class_path, action)
-        schema["properties"].update(
-            {
-                "class_path": {"const": class_path},
-                "init_args": init_args_schema,
-                "dict_kwargs": {"type": "object"},
-            }
-        )
+        class_parser = self.get_class_parser(class_path, action)
+        init_args_schema = {"type": "object"} if class_parser is None else self.parser_schema(class_parser)
+        schema["properties"].update({"class_path": {"const": class_path}, "init_args": init_args_schema})
+        # resolved parameters are only described in init_args, even though parsing also takes them from dict_kwargs
+        if class_parser is None or class_parser._accepted_kwargs[None] is True:
+            schema["properties"]["dict_kwargs"] = {"type": "object"}
         # init_args can only be omitted when none of the init parameters is required
         schema["required"] = ["class_path"] + (["init_args"] if init_args_schema.get("required") else [])
         return schema
@@ -569,14 +567,22 @@ def unknown_class_path_schema(self) -> dict:
         schema["required"] = ["class_path"]
         return schema
 
-    def class_parser_schema(self, class_type, action, description: Optional[str] = None) -> dict:
+    def get_class_parser(self, class_type, action):
         sub_add_kwargs = dict(getattr(action, "sub_add_kwargs", None) or {})
         sub_add_kwargs.pop("linked_targets", None)
         try:
-            class_parser = ActionTypeHint.get_class_parser(class_type, sub_add_kwargs=sub_add_kwargs)
+            return ActionTypeHint.get_class_parser(class_type, sub_add_kwargs=sub_add_kwargs)
         except Exception as ex:
             action.logger.debug(f"Unable to get schema for init args of '{class_type}': {ex}")
-            return {"type": "object"}
+            return None
+
+    def parser_schema(self, parser, description: Optional[str] = None) -> dict:
         schema = new_object(description)
-        self.add_properties(class_parser, schema)
+        self.add_properties(parser, schema)
         return schema
+
+    def class_parser_schema(self, class_type, action, description: Optional[str] = None) -> dict:
+        class_parser = self.get_class_parser(class_type, action)
+        if class_parser is None:
+            return {"type": "object"}
+        return self.parser_schema(class_parser, description)

jsonargparse/_core.py

@@ -138,6 +138,7 @@ class ActionsContainer(ArgumentLinking, InstantiateMethod, SignatureArguments, a
     def __init__(self, *args, **kwargs) -> None:
         """Initializer for ActionsContainer instance."""
         super().__init__(*args, **kwargs)
+        self._accepted_kwargs = {}
         self.register("type", None, identity)
         self.register("action", "parsers", ActionSubCommands)
         self.register("action", "config", ActionConfigFile)

jsonargparse/_parameter_resolvers.py

@@ -12,7 +12,7 @@
 from functools import partial, partialmethod
 from importlib import import_module
 from types import MethodType
-from typing import Any, Union
+from typing import Any, Literal, Union
 
 from ._common import (
     LoggerProperty,
@@ -267,7 +267,7 @@
     given_args = set(ast_get_call_positional_indexes(node))
     given_kwargs = set(ast_get_call_keyword_names(node))
     input_params = params
-    params = [p for n, p in enumerate(params) if n not in given_args]
+    params = [p for n, p in enumerate(params) if n not in given_args or p.kind == kinds.VAR_KEYWORD]
     params = [p for p in params if p.name not in given_kwargs]
     if removed_params is not None and len(params) < len(input_params):
         removed_params.update(p.name for p in input_params if p.name in given_kwargs)
@@ -278,6 +278,31 @@
     return next((n for n, p in enumerate(params) if p.kind == kind), -1)
 
 
+def get_unresolved_kwargs_param(component, parent) -> ParamData:
+    """Parameter that stands for a ``**kwargs`` that the resolvers were unable to resolve.
+
+    Its presence means that keyword arguments other than the resolved ones are accepted. The name
+    is not a valid identifier so that it can't collide with a real parameter.
+    """
+    return ParamData(name="**", annotation=inspect._empty, kind=kinds.VAR_KEYWORD, component=component, parent=parent)
+
+
+def accepts_unresolved_kwargs(params: ParamList) -> bool:
+    """Whether the parameters include an unresolved ``**kwargs``."""
+    return any(p.kind == kinds.VAR_KEYWORD for p in params)
+
+
+def remove_unresolved_kwargs(params: ParamList) -> ParamList:
+    return [p for p in params if p.kind != kinds.VAR_KEYWORD]
+
+
+def get_accepted_kwargs(params: ParamList) -> set[str] | Literal[True]:
+    """Names of the keyword arguments accepted, or True when any name is accepted."""
+    if accepts_unresolved_kwargs(params):
+        return True
+    return {n for p in params for n in (p.name, *(p.aliases or ()))}
+
+
 def get_signature_parameters_and_indexes(component, parent, logger):
     signature_source = component
     if is_classmethod(parent, component):
@@ -436,7 +461,7 @@
 
 def split_args_and_kwargs(params: ParamList) -> tuple[ParamList, ParamList]:
     args = [p for p in params if p.kind == kinds.POSITIONAL_ONLY]
-    kwargs = [p for p in params if p.kind in {kinds.KEYWORD_ONLY, kinds.POSITIONAL_OR_KEYWORD}]
+    kwargs = [p for p in params if p.kind in {kinds.KEYWORD_ONLY, kinds.POSITIONAL_OR_KEYWORD, kinds.VAR_KEYWORD}]
     return args, kwargs
 
 
@@ -757,7 +782,9 @@
                     get_param_args = self.get_node_component(node, source)
                     if get_param_args:
                         try:
-                            params = get_signature_parameters(*get_param_args, logger=self.logger)
+                            params = get_signature_parameters(
+                                *get_param_args, logger=self.logger, include_var_keyword=True
+                            )
                         except Exception:
                             self.log_debug(f"failed to get parameters for call that uses attr: {get_param_args}")
             params = remove_given_parameters(node, params)
@@ -835,66 +862,69 @@
             origin=param_kwargs_pop_or_get + self.get_node_origin(node),
         )
 
-    def get_parameters_args_and_kwargs(self) -> tuple[ParamList, ParamList]:
+    def get_parameters_args_and_kwargs(self, kwargs_idx: int) -> tuple[ParamList, ParamList]:
         self.parse_source_tree()
         args_name = getattr(self.component_node.args.vararg, "arg", None)
-        kwargs_name = getattr(self.component_node.args.kwarg, "arg", None)
+        # kwargs_idx is negative when the **kwargs was already replaced, i.e. an Unpack[TypedDict]
+        kwargs_name = getattr(self.component_node.args.kwarg, "arg", None) if kwargs_idx >= 0 else None
         values_to_find = {}
         if args_name:
             values_to_find[args_name] = ast_variable_load(args_name)
         if kwargs_name:
             values_to_find[kwargs_name] = ast_variable_load(kwargs_name)
 
         values_found = self.find_values_usage(values_to_find)
-        if not values_found:
-            return [], []
+        kwargs_uses = [(v, s) for k, v, s in values_found if k == kwargs_name] if kwargs_name else []
+        # a **kwargs not used in the body is discarded by the component, but still accepted
+        kwargs_unresolved = bool(kwargs_name) and not kwargs_uses
 
         params_list = []
         removed_params: set[str] = set()
         pop_or_get_params: set[str] = set()
         kwargs_value = kwargs_name and values_to_find[kwargs_name]
         kwargs_value_dump = kwargs_value and ast.dump(kwargs_value)
-        for node, source in [(v, s) for k, v, s in values_found if k == kwargs_name]:
+        for node, source in kwargs_uses:
+            params = None  # None means that where the kwargs go could not be determined
             if isinstance(node, ast.Call):
                 if ast_is_kwargs_pop_or_get(node, kwargs_value_dump):
                     param = self.get_kwargs_pop_or_get_parameter(node, self.component, self.parent, self.doc_params)
                     pop_or_get_params.add(param.name)
                     params_list.append([param])
                     continue
                 kwarg = ast_get_call_kwarg_with_value(node, kwargs_value)
-                params = []
                 if kwarg.arg:
                     self.log_debug(f"kwargs given as keyword parameter not supported: {ast.unparse(node)}")
                 elif self.parent and ast_is_super_call(node):
                     if ast_is_supported_super_call(node, self.self_name, self.log_debug):
                         params = get_mro_parameters(
                             node.func.attr,  # type: ignore[attr-defined]
-                            get_signature_parameters,
+                            partial(get_signature_parameters, include_var_keyword=True),
                             self.logger,
                         )
                 else:
                     get_param_args = self.get_node_component(node, source)
                     if get_param_args:
-                        params = get_signature_parameters(*get_param_args, logger=self.logger)
-                params = remove_given_parameters(node, params, removed_params)
-                if params:
-                    self.add_node_origins(params, node)
-                    params_list.append(params)
-            elif isinstance(node, ast_assign_type):
-                self_attr = self.parent and ast_is_attr_assign(node, self.self_name)
-                if self_attr:
-                    params = self.get_parameters_attr_use_in_members(self_attr)
-                    if params:
-                        self.add_node_origins(params, node)
-                        params_list.append(params)
-                else:
-                    self.log_debug(f"unsupported type of assign: {ast.unparse(node)}")
+                        params = get_signature_parameters(*get_param_args, logger=self.logger, include_var_keyword=True)
+                if params is not None:
+                    params = remove_given_parameters(node, params, removed_params)
+            elif self.parent and (self_attr := ast_is_attr_assign(node, self.self_name)):
+                params = self.get_parameters_attr_use_in_members(self_attr) or None
+            else:
+                self.log_debug(f"unsupported type of assign: {ast.unparse(node)}")
+            if params is None:
+                kwargs_unresolved = True
+            elif params:
+                self.add_node_origins(params, node)
+                params_list.append(params)
 
         params = group_parameters(params_list)
         # a pop/get from kwargs means the parameter is accepted, even if the value is then given explicitly
         removed_params -= pop_or_get_params
         params = [p for p in params if p.name not in removed_params]
-        return split_args_and_kwargs(params)
+        args, kwargs = split_args_and_kwargs(params)
+        if kwargs_unresolved and not accepts_unresolved_kwargs(kwargs):
+            kwargs.append(get_unresolved_kwargs_param(self.component, self.parent))
+        return args, kwargs
 
     def get_parameters_attr_use_in_members(self, attr_name) -> ParamList:
         attr_value = ast_attribute_load(self.self_name, attr_name)
@@ -955,7 +985,7 @@
         if args_idx >= 0 or kwargs_idx >= 0:
             self.doc_params = doc_params
             with mro_context(self.parent):
-                args, kwargs = self.get_parameters_args_and_kwargs()
+                args, kwargs = self.get_parameters_args_and_kwargs(kwargs_idx)
             params = replace_args_and_kwargs(params, args, kwargs)
         add_stub_types(stubs, params, self.component)
         params = self.remove_ignore_parameters(params)
@@ -1198,7 +1228,8 @@
     if stub_import:
         origin = get_parameter_origins(component, parent)
         aliases = resolver.get_aliases(stub_import)
-        arg_asts = stub_import.info.ast.args.args + stub_import.info.ast.args.kwonlyargs
+        args_ast = stub_import.info.ast.args
+        arg_asts = args_ast.args + args_ast.kwonlyargs
         params = []
         for num, arg_ast in enumerate(arg_asts):
             if parent and num == 0:
@@ -1218,6 +1249,8 @@
                     origin=origin,
                 )
             )
+        if args_ast.kwarg:
+            params.append(get_unresolved_kwargs_param(component, parent))
     return params
 
 
@@ -1236,7 +1269,7 @@
             args, kwargs = split_args_and_kwargs(subparams)
             params = replace_args_and_kwargs(params, args, kwargs)
 
-    params = replace_args_and_kwargs(params, [], [])
+    params = replace_args_and_kwargs(params, [], [get_unresolved_kwargs_param(component, parent)])
     add_stub_types(stubs, params, component)
     return params
 
@@ -1245,6 +1278,7 @@
     function_or_class: Callable | type,
     method_or_property: str | None = None,
     logger: bool | str | dict | logging.Logger = True,
+    include_var_keyword: bool = False,
 ) -> ParamList:
     """Get parameters by inspecting ASTs, stubs or by inheritance assumptions.
 
@@ -1258,6 +1292,8 @@
             which to get the signature parameters. If not provided it returns
             the parameters for ``__init__``.
         logger: Useful for debugging. Only logs at ``DEBUG`` level.
+        include_var_keyword: Whether to include a ``**kwargs`` that could not be resolved, see
+            :func:`get_unresolved_kwargs_param`.
     """
     from ._typehints import is_namedtuple, is_typed_dict
 
@@ -1295,4 +1331,6 @@
         attr = inspect.getattr_static(get_generic_origin(parent), method_name)
         if is_partial_method(attr) and component is attr.func:
             params = apply_partial_method(params, attr)
+    if not include_var_keyword:
+        params = remove_unresolved_kwargs(params)
     return params