From b4b391482a01dc8bd3d036193fe479e07e994c7d Mon Sep 17 00:00:00 2001 From: WEN Hao Date: Thu, 5 Nov 2026 00:19:14 +0800 Subject: [PATCH] docs(zh_CN): fix the three inconsistent-reference warnings in the zh build Sphinx 9's stricter i18n reference check flags zh translations whose raw references differ from the English originals: - the SCAFFOLD and convergence-theorem link texts in the algorithm theory pages keep the original English raw reference form (proper-noun link labels); - the optimizers module docstring translation restores the two :mod: references (torch.optim, torch_optimizer) it had dropped. The zh_CN build returns to zero warnings. --- .../zh_CN/LC_MESSAGES/algorithms/overview.po | 142 ++++++++++++++---- .../zh_CN/LC_MESSAGES/algorithms/proximal.po | 101 ++++++++++--- .../zh_CN/LC_MESSAGES/api/optimizers.po | 17 +-- 3 files changed, 196 insertions(+), 64 deletions(-) diff --git a/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/overview.po b/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/overview.po index 69c45796..f2a5e82b 100644 --- a/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/overview.po +++ b/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/overview.po @@ -24,7 +24,9 @@ msgid "Overview of Optimization Algorithms in Federated Learning" msgstr "联邦学习中的优化算法概述" #: ../../source/algorithms/overview.rst:6 -msgid "The most important contribution of the initial work on federated learning [:footcite:ct:`mcmahan2017fed_avg`] was the introduction of the Federated Averaging algorithm (``FedAvg``). Mathematically, federated learning solves the following problem of minimization of empirical risk function" +msgid "" +"The most important contribution of the initial work on federated learning [:footcite:ct:`mcmahan2017fed_avg`] was the introduction of the Federated Averaging algorithm (``FedAvg``). Mathematically, " +"federated learning solves the following problem of minimization of empirical risk function" msgstr "联邦学习开创性的文献 [:footcite:ct:`mcmahan2017fed_avg`] 中，最重要的贡献便是联邦平均算法（ ``FedAvg`` ）的提出。数学上，联邦学习求解如下极小化经验风险函数的问题" #: ../../source/algorithms/overview.rst:10 @@ -36,7 +38,9 @@ msgid "" msgstr "" #: ../../source/algorithms/overview.rst:18 -msgid "where :math:`\\ell_k` is the loss function of client :math:`k`, :math:`\\mathcal{D}_k` is the data distribution of client :math:`k`, :math:`\\mathcal{P}` is the distribution of clients, and :math:`\\mathbb{E}` is the expectation operator. If we simply let :math:`\\mathcal{P} = \\{1, 2, \\ldots, K\\}`, then the optimization problem can be simplified as" +msgid "" +"where :math:`\\ell_k` is the loss function of client :math:`k`, :math:`\\mathcal{D}_k` is the data distribution of client :math:`k`, :math:`\\mathcal{P}` is the distribution of clients, and " +":math:`\\mathbb{E}` is the expectation operator. If we simply let :math:`\\mathcal{P} = \\{1, 2, \\ldots, K\\}`, then the optimization problem can be simplified as" msgstr "其中 :math:`\\ell_k` 是客户端 :math:`k` 的损失函数，:math:`\\mathcal{D}_k` 是客户端 :math:`k` 的数据分布，:math:`\\mathcal{P}` 是客户端的分布，:math:`\\mathbb{E}` 是期望算子。假设我们令 :math:`\\mathcal{P} = \\{1, 2, \\ldots, K\\}` ，则上述优化问题可以简记为" #: ../../source/algorithms/overview.rst:23 @@ -124,8 +128,14 @@ msgid "" msgstr "" #: ../../source/algorithms/overview.rst:92 -msgid "(A4-2) Data distribution is non-I.I.D across clients, in which case we need a quantity to measure the degree of this statistical heterogeneity. This quantity can be defined in a number of ways [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]. For example, in [:footcite:ct:`karimireddy2020scaffold`] and [:footcite:ct:`zhang2020fedpd`], the so-called bounded gradient dissimilarity (BGD), denoted as :math:`(G; B)`-BGD, is used as this quantity. More specifically, there exists constants :math:`G > 0` and :math:`B \\geqslant 0` such that" -msgstr "(A4-2) 数据非独立同分布 (Non-I.I.D.)，这时候我们需要有一个量，用来度量这种统计上的异质性的程度。这个量可以有多种定义方式 [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]，这里采用 [:footcite:ct:`karimireddy2020scaffold`] 与 [:footcite:ct:`zhang2020fedpd`] 中定义的梯度差异有界性 (Bounded Gradient Dissimilarity, BGD)，记作 :math:`(G; B)`-BGD。具体来说，存在常数 :math:`G > 0` 以及 :math:`B \\geqslant 0` ，满足" +msgid "" +"(A4-2) Data distribution is non-I.I.D across clients, in which case we need a quantity to measure the degree of this statistical heterogeneity. This quantity can be defined in a number of ways " +"[:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]. For example, in [:footcite:ct:`karimireddy2020scaffold`] and [:footcite:ct:`zhang2020fedpd`], the so-" +"called bounded gradient dissimilarity (BGD), denoted as :math:`(G; B)`-BGD, is used as this quantity. More specifically, there exists constants :math:`G > 0` and :math:`B \\geqslant 0` such that" +msgstr "" +"(A4-2) 数据非独立同分布 (Non-I.I.D.)，这时候我们需要有一个量，用来度量这种统计上的异质性的程度。这个量可以有多种定义方式 [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]，这里采用 " +"[:footcite:ct:`karimireddy2020scaffold`] 与 [:footcite:ct:`zhang2020fedpd`] 中定义的梯度差异有界性 (Bounded Gradient Dissimilarity, BGD)，记作 :math:`(G; B)`-BGD。具体来说，存在常数 :math:`G > 0` 以及 :math:`B \\geqslant 0` " +"，满足" #: ../../source/algorithms/overview.rst:99 msgid "\\dfrac{1}{K} \\sum\\limits_{k=1}^K \\lVert \\nabla f_k(\\theta) \\rVert^2 \\leqslant G^2 + B^2 \\lVert \\nabla f(\\theta) \\rVert^2, ~ \\forall \\theta \\in \\mathbb{R}^d." @@ -154,7 +164,9 @@ msgid "where :math:`\\mathcal{C}` is a convex set on which :math:`f` is defined. msgstr "其中 :math:`\\mathcal{C}` 是一个凸集，:math:`f` 定义在其上。" #: ../../source/algorithms/overview.rst:120 -msgid "(A5-2) Strong convexity: there exists a constant :math:`\\mu > 0` such that :math:`f - \\frac{\\mu}{2} \\lVert \\theta \\rVert^2` is convex. In this case, we say that :math:`f` is :math:`\\mu`-strongly convex." +msgid "" +"(A5-2) Strong convexity: there exists a constant :math:`\\mu > 0` such that :math:`f - \\frac{\\mu}{2} \\lVert \\theta \\rVert^2` is convex. In this case, we say that :math:`f` is " +":math:`\\mu`-strongly convex." msgstr "(A5-2) 强凸性：:math:`\\mu > 0` ，若 :math:`f - \\frac{\\mu}{2} \\lVert \\theta \\rVert^2` 是凸函数，则称 :math:`f` 是 :math:`\\mu`-强凸函数。" #: ../../source/algorithms/overview.rst:123 @@ -170,27 +182,44 @@ msgid "" msgstr "" #: ../../source/algorithms/overview.rst:134 -msgid "It is easy to find the equivalence between the constrained optimization problem :eq:`fl-basic-constraint` and the unconstrained optimization problem :eq:`fl-basic`. The constrained formulation :eq:`fl-basic-constraint` is called the **consensus problem** in the literature of distributed optimization [:footcite:ct:`boyd2011distributed`]. The superiority of the constrained formulation :eq:`fl-basic-constraint` is that the objective function becomes block-separable, which is more suitable for the design of parallel and distributed algorithms." -msgstr "我们很容易看出约束优化问题 :eq:`fl-basic-constraint` 与无约束优化问题 :eq:`fl-basic` 两种格式的等价性。格式 :eq:`fl-basic-constraint` 在分布式优化中被称作共识优化问题 (Consensus Problem) [:footcite:ct:`boyd2011distributed`]。约束格式 :eq:`fl-basic-constraint` 的好处是，目标函数变成了分块可分离的 (Block Separable)，这对设计可并行化的算法是巨大的便利。" +msgid "" +"It is easy to find the equivalence between the constrained optimization problem :eq:`fl-basic-constraint` and the unconstrained optimization problem :eq:`fl-basic`. The constrained formulation " +":eq:`fl-basic-constraint` is called the **consensus problem** in the literature of distributed optimization [:footcite:ct:`boyd2011distributed`]. The superiority of the constrained formulation " +":eq:`fl-basic-constraint` is that the objective function becomes block-separable, which is more suitable for the design of parallel and distributed algorithms." +msgstr "" +"我们很容易看出约束优化问题 :eq:`fl-basic-constraint` 与无约束优化问题 :eq:`fl-basic` 两种格式的等价性。格式 :eq:`fl-basic-constraint` 在分布式优化中被称作共识优化问题 (Consensus Problem) [:footcite:ct:`boyd2011distributed`]。约束格式 :eq:`fl-basic-" +"constraint` 的好处是，目标函数变成了分块可分离的 (Block Separable)，这对设计可并行化的算法是巨大的便利。" #: ../../source/algorithms/overview.rst:140 msgid "Federated Averaging Algorithm" msgstr "联邦平均算法" #: ../../source/algorithms/overview.rst:142 -msgid "The core idea of the ``FedAvg`` algorithm is to make full use of the local computation resources of each client so that each client can perform multiple local iterations before uploading the local model to the server. It alleviates the problem of straggler clients and reduces the communication overhead, hence accelerating the convergence of the algorithm. This may well be thought of as a simple form of **skipping** algorithm, which were further developed in [:footcite:ct:`zhang2020fedpd, proxskip, proxskip-vr`]. Pseudocode for ``FedAvg`` is shown as follows:" -msgstr "联邦平均算法的核心思想，是充分利用子节点的计算能力，在以学习率 :math:`\\eta` 执行完一定轮数的随机梯度下降 (Stochastic Gradient Descent, SGD) 之后，由中心节点收集各子节点发送过来的模型参数，进行平均 (Averaging)，之后再将平均之后的模型参数广播 (Broadcast) 给子节点进行下一轮迭代。这样规避了每一轮 SGD 都进行通信带来的巨大的通信开销，在加速模型训练的同时，数值上在某些情况下还有更好的收敛性。这实际上是一种比较朴素与简单的“跳步”的思想，这种思想在随后的研究 [:footcite:ct:`zhang2020fedpd, proxskip, proxskip-vr`] 中也得到了进一步的发展。 ``FedAvg`` 的伪代码如下：" +msgid "" +"The core idea of the ``FedAvg`` algorithm is to make full use of the local computation resources of each client so that each client can perform multiple local iterations before uploading the local " +"model to the server. It alleviates the problem of straggler clients and reduces the communication overhead, hence accelerating the convergence of the algorithm. This may well be thought of as a " +"simple form of **skipping** algorithm, which were further developed in [:footcite:ct:`zhang2020fedpd, proxskip, proxskip-vr`]. Pseudocode for ``FedAvg`` is shown as follows:" +msgstr "" +"联邦平均算法的核心思想，是充分利用子节点的计算能力，在以学习率 :math:`\\eta` 执行完一定轮数的随机梯度下降 (Stochastic Gradient Descent, SGD) 之后，由中心节点收集各子节点发送过来的模型参数，进行平均 (Averaging)，之后再将平均之后的模型参数广播 (Broadcast) 给子节点进行下一轮迭代。这样规避了每一轮 SGD " +"都进行通信带来的巨大的通信开销，在加速模型训练的同时，数值上在某些情况下还有更好的收敛性。这实际上是一种比较朴素与简单的“跳步”的思想，这种思想在随后的研究 [:footcite:ct:`zhang2020fedpd, proxskip, proxskip-vr`] 中也得到了进一步的发展。 ``FedAvg`` 的伪代码如下：" #: ../../source/algorithms/overview.rst:153 -msgid "``FedAvg`` achieved some good numerical results (see Section 3 of [:footcite:ct:`mcmahan2017fed_avg`]), but its convergence, espcially under non-I.I.D. data distributions, is not properly analyzed (see [:footcite:ct:`khaled2019_first, Khaled2020_tighter`]). There are several works that deal with this issue (such as [:footcite:ct:`zhou_2018_convergence, li2019convergence`]) with extra assumptions such as the convexity of the objective function :math:`f`, etc." -msgstr "联邦平均算法 ``FedAvg`` 在数值上取得了一些不错的实验结果（见 [:footcite:ct:`mcmahan2017fed_avg`] 第 3 节），但是其收敛性，特别是在数据非独立同分布的情况下，并未得到很好的分析（见 [:footcite:ct:`khaled2019_first, Khaled2020_tighter`]）。后续的文献（如 [:footcite:ct:`zhou_2018_convergence, li2019convergence`]）处理了这一问题，但添加了诸如 :math:`f` 强凸等额外假设。" +msgid "" +"``FedAvg`` achieved some good numerical results (see Section 3 of [:footcite:ct:`mcmahan2017fed_avg`]), but its convergence, espcially under non-I.I.D. data distributions, is not properly analyzed " +"(see [:footcite:ct:`khaled2019_first, Khaled2020_tighter`]). There are several works that deal with this issue (such as [:footcite:ct:`zhou_2018_convergence, li2019convergence`]) with extra " +"assumptions such as the convexity of the objective function :math:`f`, etc." +msgstr "" +"联邦平均算法 ``FedAvg`` 在数值上取得了一些不错的实验结果（见 [:footcite:ct:`mcmahan2017fed_avg`] 第 3 节），但是其收敛性，特别是在数据非独立同分布的情况下，并未得到很好的分析（见 [:footcite:ct:`khaled2019_first, Khaled2020_tighter`]）。后续的文献（如 " +"[:footcite:ct:`zhou_2018_convergence, li2019convergence`]）处理了这一问题，但添加了诸如 :math:`f` 强凸等额外假设。" #: ../../source/algorithms/overview.rst:160 msgid "``FedAvg`` from the Perspective of Optimization" msgstr "最优化理论视角下的联邦平均算法" #: ../../source/algorithms/overview.rst:162 -msgid "In this section, we will analyze the ``FedAvg`` algorithm from the perspective of optimization theory. In fact, the optimization problem :eq:`fl-basic` that ``FedAvg`` solves can be equivalently reformulated as the following constrained optimization problem:" +msgid "" +"In this section, we will analyze the ``FedAvg`` algorithm from the perspective of optimization theory. In fact, the optimization problem :eq:`fl-basic` that ``FedAvg`` solves can be equivalently " +"reformulated as the following constrained optimization problem:" msgstr "我们从最优化理论的角度对联邦平均算法 ``FedAvg`` 进行分析。事实上， ``FedAvg`` 所处理的优化问题 :eq:`fl-basic` 可以等价地改写为如下的约束优化问题：" #: ../../source/algorithms/overview.rst:166 @@ -204,15 +233,23 @@ msgid "" msgstr "" #: ../../source/algorithms/overview.rst:176 -msgid "where :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\mathbb{R}^d` and :math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` is a convex set in :math:`\\mathbb{R}^{Kd}`. Projected gradient descent (PGD) is an effective method for solving the constrained optimization problem :eq:`fedavg-constraint`, which has the following update rule:" -msgstr "其中 :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\mathbb{R}^d` ，:math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` 是 :math:`\\mathbb{R}^{Kd}` 中一个凸集。我们知道，投影梯度下降法 (Projected Gradient Descent, PGD) 是求解问题 :eq:`fedavg-constraint` 的一个有效算法，其迭代格式为：" +msgid "" +"where :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\mathbb{R}^d` and " +":math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` is a convex set in :math:`\\mathbb{R}^{Kd}`. Projected gradient descent (PGD) is an effective method" +" for solving the constrained optimization problem :eq:`fedavg-constraint`, which has the following update rule:" +msgstr "" +"其中 :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\mathbb{R}^d` " +"，:math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` 是 :math:`\\mathbb{R}^{Kd}` 中一个凸集。我们知道，投影梯度下降法 (Projected Gradient Descent, PGD) 是求解问题 :eq:`fedavg-" +"constraint` 的一个有效算法，其迭代格式为：" #: ../../source/algorithms/overview.rst:181 msgid "\\Theta^{(t+1)} = \\Pi_{\\mathcal{E}} \\left( \\Theta^{(t)} - \\eta \\nabla F(\\Theta^{(t)}) \\right)," msgstr "" #: ../../source/algorithms/overview.rst:186 -msgid "where :math:`\\Pi_{\\mathcal{E}}` is the projection operator onto the set :math:`\\mathcal{E}`. It is easy to show that the projection operator onto the set :math:`\\mathcal{E}` is indeed the average operator, i.e.," +msgid "" +"where :math:`\\Pi_{\\mathcal{E}}` is the projection operator onto the set :math:`\\mathcal{E}`. It is easy to show that the projection operator onto the set :math:`\\mathcal{E}` is indeed the " +"average operator, i.e.," msgstr "其中 :math:`\\Pi_{\\mathcal{E}}` 是到集合 :math:`\\mathcal{E}` 的投影算子。容易看出，到集合 :math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` 的投影映射就是“求平均”映射：" #: ../../source/algorithms/overview.rst:189 @@ -220,7 +257,9 @@ msgid "\\Pi_{\\mathcal{E}}: \\mathbb{R}^{Kd} \\to \\mathcal{E}: ( \\theta_1, \\l msgstr "" #: ../../source/algorithms/overview.rst:194 -msgid "We have shown that mathematically the ``FedAvg`` algorithm is indeed a kind of stochastic projected gradient descent (SPGD) algorithm, where the clients perform local stochastic gradient descent (SGD) updates and the server performs the projection step :eq:`fedavg-projection`." +msgid "" +"We have shown that mathematically the ``FedAvg`` algorithm is indeed a kind of stochastic projected gradient descent (SPGD) algorithm, where the clients perform local stochastic gradient descent " +"(SGD) updates and the server performs the projection step :eq:`fedavg-projection`." msgstr "也就是说，联邦平均算法在数学上，实质上是一种随机投影梯度法：子节点执行的是梯度步（具体来说是用的随机梯度下降），中心节点执行的是投影步 :eq:`fedavg-projection` 。" #: ../../source/algorithms/overview.rst:199 @@ -228,11 +267,19 @@ msgid "A Direct Improvement of ``FedAvg``" msgstr "联邦平均算法的改进" #: ../../source/algorithms/overview.rst:201 -msgid "Since ``FedAvg`` is based on stochastic gradient descent (SGD), it is natural to consider applying acceleration techniques [:footcite:ct:`adagrad, adam, Zaheer_2018_yogi, adamw_amsgrad`] to improve the algorithm performance. Computation on clients and on the server are relatively decoupled, so it does not require large modifications to the whole algorithm framework. Indeed, the authors of the ``FedAvg`` paper put this idea into practice and proposed a federated learning framework called ``FedOpt`` [:footcite:ct:`reddi2020fed_opt`] which has stronger adaptability. The pseudocode for ``FedOpt`` is shown as follows:" -msgstr "因为联邦平均算法是以随机梯度下降法为基础的，那么一个自然的想法就是将随机梯度下降法的一些加速的技巧 [:footcite:ct:`adagrad, adam, Zaheer_2018_yogi, adamw_amsgrad`] 引入联邦学习整个算法体系内。由于联邦学习子节点与中心节点的计算相对独立、没有耦合，所以这种做法并不需要对联邦学习整个算法框架做大改动，是一种可行的方法。事实上，联邦平均算法的作者在随后发表的文章 [:footcite:ct:`reddi2020fed_opt`] 中实践了这一想法，提出了一个泛用性更强的自适应联邦学习算法框架 ``FedOpt`` ，其伪代码如下：" +msgid "" +"Since ``FedAvg`` is based on stochastic gradient descent (SGD), it is natural to consider applying acceleration techniques [:footcite:ct:`adagrad, adam, Zaheer_2018_yogi, adamw_amsgrad`] to improve " +"the algorithm performance. Computation on clients and on the server are relatively decoupled, so it does not require large modifications to the whole algorithm framework. Indeed, the authors of the " +"``FedAvg`` paper put this idea into practice and proposed a federated learning framework called ``FedOpt`` [:footcite:ct:`reddi2020fed_opt`] which has stronger adaptability. The pseudocode for " +"``FedOpt`` is shown as follows:" +msgstr "" +"因为联邦平均算法是以随机梯度下降法为基础的，那么一个自然的想法就是将随机梯度下降法的一些加速的技巧 [:footcite:ct:`adagrad, adam, Zaheer_2018_yogi, adamw_amsgrad`] " +"引入联邦学习整个算法体系内。由于联邦学习子节点与中心节点的计算相对独立、没有耦合，所以这种做法并不需要对联邦学习整个算法框架做大改动，是一种可行的方法。事实上，联邦平均算法的作者在随后发表的文章 [:footcite:ct:`reddi2020fed_opt`] 中实践了这一想法，提出了一个泛用性更强的自适应联邦学习算法框架 ``FedOpt`` ，其伪代码如下：" #: ../../source/algorithms/overview.rst:212 -msgid "In the above pseudocode, :math:`\\operatorname{aggregate} \\left( \\left\\{ \\Delta_{k}^{(t)} \\right\\}_{k \\in \\mathcal{S}^{(t)}} \\right)` refers to some method that aggregates the local inertia updates :math:`\\Delta_{k}^{(t)}` from the selected clients :math:`\\mathcal{S}^{(t)}` into a global inertia update :math:`\\Delta^{(t)}`. This method, for example, can be simply averaging" +msgid "" +"In the above pseudocode, :math:`\\operatorname{aggregate} \\left( \\left\\{ \\Delta_{k}^{(t)} \\right\\}_{k \\in \\mathcal{S}^{(t)}} \\right)` refers to some method that aggregates the local inertia" +" updates :math:`\\Delta_{k}^{(t)}` from the selected clients :math:`\\mathcal{S}^{(t)}` into a global inertia update :math:`\\Delta^{(t)}`. This method, for example, can be simply averaging" msgstr "这里的 :math:`\\operatorname{aggregate} \\left( \\left\\{ \\Delta_{k}^{(t)} \\right\\}_{k \\in \\mathcal{S}^{(t)}} \\right)` 指的是将来自子节点的惯性项 :math:`\\Delta_{k}^{(t)}` 聚合为全局惯性量的方法，例如可以采用简单的均值" #: ../../source/algorithms/overview.rst:216 @@ -248,7 +295,9 @@ msgid "\\Delta^{(t)} \\gets \\beta_1 \\Delta^{(t-1)} + \\left( (1 - \\beta_1) / msgstr "" #: ../../source/algorithms/overview.rst:228 -msgid "As one has already noticed, compared to ``FedAvg``, ``FedOpt`` introduces some momentum terms on the server node (in **ServerOpt**) to accelerate the convergence. In [:footcite:ct:`reddi2020fed_opt`], the authors listed several options for **ServerOpt**:" +msgid "" +"As one has already noticed, compared to ``FedAvg``, ``FedOpt`` introduces some momentum terms on the server node (in **ServerOpt**) to accelerate the convergence. In " +"[:footcite:ct:`reddi2020fed_opt`], the authors listed several options for **ServerOpt**:" msgstr "我们注意到，相比于联邦平均算法 ``FedAvg`` ，自适应联邦学习算法 ``FedOpt`` 在中心节点引入了带动量 (Momentum) 的项，即 **ServerOpt** ，用于加速。我们列举一些 **ServerOpt** 可供选择的方法 [:footcite:ct:`reddi2020fed_opt`]：" #: ../../source/algorithms/overview.rst:231 @@ -288,16 +337,31 @@ msgid "" msgstr "" #: ../../source/algorithms/overview.rst:261 -msgid "``FedOpt`` applys acceleration techniques which are frequently used in general machine learning tasks to the field of federated learning. It is a direct improvement of ``FedAvg`` which is simple but important. Moreover, it demonstrates the decoupling of the computation on clients and on the server, which is a key feature of federated learning." +msgid "" +"``FedOpt`` applys acceleration techniques which are frequently used in general machine learning tasks to the field of federated learning. It is a direct improvement of ``FedAvg`` which is simple but" +" important. Moreover, it demonstrates the decoupling of the computation on clients and on the server, which is a key feature of federated learning." msgstr "``FedOpt`` 将机器学习领域内常用的一些加速的技巧应用于联邦学习的场景，是 ``FedAvg`` 较为平凡但又重要的推广。它另一个更重要的意义在于充分表明了联邦学习子节点与中心节点的计算的解耦合的性质。" #: ../../source/algorithms/overview.rst:265 -msgid "To better handle non-I.I.D. data, one needs to introduce some other techniques. In non-I.I.D. scenarios, the gradients have different distributions across clients. A natural idea is to bring in some extra parameters which update along with the model parameters to make corrections (modifications) to the gradients on clients, reducing their variance and further accelerating the convergence. This technique is the so-called **variance reduction** technique [:footcite:ct:`johnson2013accelerating`], which was first introduced to federated learning in [:footcite:ct:`karimireddy2020scaffold`] in the form of a new federated learning algorithm called **SCAFFOLD** (Stochastic Controlled Averaging algorithm). The pseudocode for **SCAFFOLD** is shown as follows:" -msgstr "联邦平均算法 ``FedAvg`` 与自适应联邦学习算法 ``FedOpt`` 考虑的主要是数据在各子节点之间独立同分布的场景。对于更具有实际意义的数据非独立同分布的场景，我们需要引入一些别的工具来处理这种情况。此时，各子节点上的梯度在分布上是有差异的，一个很自然的想法就是引入一些额外的参数，与模型参数一起更新，利用这些额外的参数对子节点的梯度进行修正，减小各子节点上的梯度在分布上的差异，进而达到使算法收敛更稳定以及加速收敛的目的。这种技术就是方差缩减 (Variance Reduction, VR) 技术 [:footcite:ct:`johnson2013accelerating`]，由 [:footcite:ct:`karimireddy2020scaffold`] 以算法 **SCAFFOLD** (Stochastic Controlled Averaging algorithm) 的形式首次引入联邦学习中。 **SCAFFOLD** 的伪代码如下：" +msgid "" +"To better handle non-I.I.D. data, one needs to introduce some other techniques. In non-I.I.D. scenarios, the gradients have different distributions across clients. A natural idea is to bring in some" +" extra parameters which update along with the model parameters to make corrections (modifications) to the gradients on clients, reducing their variance and further accelerating the convergence. This" +" technique is the so-called **variance reduction** technique [:footcite:ct:`johnson2013accelerating`], which was first introduced to federated learning in [:footcite:ct:`karimireddy2020scaffold`] in" +" the form of a new federated learning algorithm called **SCAFFOLD** (Stochastic Controlled Averaging algorithm). The pseudocode for **SCAFFOLD** is shown as follows:" +msgstr "" +"联邦平均算法 ``FedAvg`` 与自适应联邦学习算法 ``FedOpt`` " +"考虑的主要是数据在各子节点之间独立同分布的场景。对于更具有实际意义的数据非独立同分布的场景，我们需要引入一些别的工具来处理这种情况。此时，各子节点上的梯度在分布上是有差异的，一个很自然的想法就是引入一些额外的参数，与模型参数一起更新，利用这些额外的参数对子节点的梯度进行修正，减小各子节点上的梯度在分布上的差异，进而达到使算法收敛更稳定以及加速收敛的目的。这种技术就是方差缩减 (Variance" +" Reduction, VR) 技术 [:footcite:ct:`johnson2013accelerating`]，由 [:footcite:ct:`karimireddy2020scaffold`] 以算法 **SCAFFOLD** (Stochastic Controlled Averaging algorithm) 的形式首次引入联邦学习中。 **SCAFFOLD** 的伪代码如下：" #: ../../source/algorithms/overview.rst:277 -msgid "Variance reduction is a technique that can be flexibly combined with most algorithms and has been widely used in federated learning for dealing with statistical heterogeneity. However, it should be noted in the `SCAFFOLD algorithm `_ that on both the server and the clients, there are extra parameters :math:`c` and :math:`c_k` to maintain, which may increase the communication cost. In scenarios which are sensitive to communication cost, this would potentially be a problem. Therefore, a better solution could be a combination of the variance reduction technique and some **skipping** techniques (e.g. [:footcite:ct:`proxskip-vr`]), which will be introduced in next sections." -msgstr "方差缩减是一种可以灵活地与大多数算法结合的技术，已在联邦学习中被广泛用于处理统计异质性。但要注意，在 `SCAFFOLD 算法 `_ 中，服务器与客户端上都有需要维护的额外参数 :math:`c` 与 :math:`c_k` ，这可能增加通信开销。在对通信开销敏感的场景中，这可能会成为问题。因此，更好的方案可能是把方差缩减技术与某种 **跳步** 技巧相结合（如 [:footcite:ct:`proxskip-vr`]），这将在后面几节介绍。" +msgid "" +"Variance reduction is a technique that can be flexibly combined with most algorithms and has been widely used in federated learning for dealing with statistical heterogeneity. However, it should be " +"noted in the `SCAFFOLD algorithm `_ that on both the server and the clients, there are extra parameters :math:`c` and :math:`c_k` to maintain, which may increase the communication " +"cost. In scenarios which are sensitive to communication cost, this would potentially be a problem. Therefore, a better solution could be a combination of the variance reduction technique and some " +"**skipping** techniques (e.g. [:footcite:ct:`proxskip-vr`]), which will be introduced in next sections." +msgstr "" +"方差缩减是一种可以灵活地与大多数算法结合的技术，已在联邦学习中被广泛用于处理统计异质性。但要注意，在 `SCAFFOLD algorithm `_ 中，服务器与客户端上都有需要维护的额外参数 :math:`c` 与 :math:`c_k` ，这可能增加通信开销。在对通信开销敏感的场景中，这可能会成为问题。因此，更好的方案可能是把方差缩减技术与某种 **跳步**" +" 技巧相结合（如 [:footcite:ct:`proxskip-vr`]），这将在后面几节介绍。" #: ../../foot_bibliography_header:1 msgid "References" @@ -357,8 +421,13 @@ msgstr "参考文献" #~ msgid "(A4-2) Data distribution is non-I.I.D across clients, in which case we need a quantity to measure" #~ msgstr "(A4-2) 数据非独立同分布 (Non-I.I.D.)，这时候我们需要有一个量，用来度量" -#~ msgid "the degree of this statistical heterogeneity. This quantity can be defined in a number of ways [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]. For example, in [:footcite:ct:`karimireddy2020scaffold`] and [:footcite:ct:`zhang2020fedpd`], the so-called bounded gradient dissimilarity (BGD), denoted as :math:`(G; B)`-BGD, is used as this quantity. More specifically, there exists constants :math:`G > 0` and :math:`B \\geqslant 0` such that" -#~ msgstr "这种统计上的异质性的程度。这个量可以有多种定义方式 [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]，这里采用 [:footcite:ct:`karimireddy2020scaffold`] 与 [:footcite:ct:`zhang2020fedpd`] 中定义的梯度差异有界性 (Bounded Gradient Dissimilarity, BGD)，记作 :math:`(G; B)`-BGD。具体来说，存在常数 :math:`G > 0` 以及 :math:`B \\geqslant 0` ，满足" +#~ msgid "" +#~ "the degree of this statistical heterogeneity. This quantity can be defined in a number of ways [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]. For " +#~ "example, in [:footcite:ct:`karimireddy2020scaffold`] and [:footcite:ct:`zhang2020fedpd`], the so-called bounded gradient dissimilarity (BGD), denoted as :math:`(G; B)`-BGD, is used as this quantity." +#~ " More specifically, there exists constants :math:`G > 0` and :math:`B \\geqslant 0` such that" +#~ msgstr "" +#~ "这种统计上的异质性的程度。这个量可以有多种定义方式 [:footcite:ct:`karimireddy2020scaffold, zhang2020fedpd, li2019convergence, sahu2018fedprox`]，这里采用 [:footcite:ct:`karimireddy2020scaffold`] 与 [:footcite:ct:`zhang2020fedpd`]" +#~ " 中定义的梯度差异有界性 (Bounded Gradient Dissimilarity, BGD)，记作 :math:`(G; B)`-BGD。具体来说，存在常数 :math:`G > 0` 以及 :math:`B \\geqslant 0` ，满足" #~ msgid "\\dfrac{1}{K} \\sum\\limits_{k=1}^K \\lVert \\nabla f_k(\\theta) \\rVert^2 \\leqslant G^2 + B^2 \\lVert \\nabla f(\\theta) \\rVert^2, ~ \\forall \\theta \\in \\R^d." #~ msgstr "" @@ -388,8 +457,13 @@ msgstr "参考文献" #~ "\\end{array}" #~ msgstr "" -#~ msgid "where :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\R^d` and :math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` is a convex set in :math:`\\R^{Kd}`. Projected gradient descent (PGD) is an effective method for solving the constrained optimization problem :eq:`fedavg-constraint`, which has the following update rule:" -#~ msgstr "其中 :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\R^d` ，:math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` 是 :math:`\\R^{Kd}` 中一个凸集。我们知道，投影梯度下降法 (Projected Gradient Descent, PGD) 是求解问题 :eq:`fedavg-constraint` 的一个有效算法，其迭代格式为：" +#~ msgid "" +#~ "where :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\R^d` and " +#~ ":math:`\\mathcal{E} = \\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` is a convex set in :math:`\\R^{Kd}`. Projected gradient descent (PGD) is an effective method for " +#~ "solving the constrained optimization problem :eq:`fedavg-constraint`, which has the following update rule:" +#~ msgstr "" +#~ "其中 :math:`\\Theta = \\col(\\theta_1, \\cdots, \\theta_K) := \\begin{pmatrix} \\theta_1 \\\\ \\vdots \\\\ \\theta_K \\end{pmatrix}, \\theta_1, \\ldots, \\theta_K \\in \\R^d` ，:math:`\\mathcal{E} = " +#~ "\\left\\{ \\Theta ~ \\middle| ~ \\theta_1 = \\cdots = \\theta_K \\right\\}` 是 :math:`\\R^{Kd}` 中一个凸集。我们知道，投影梯度下降法 (Projected Gradient Descent, PGD) 是求解问题 :eq:`fedavg-constraint` 的一个有效算法，其迭代格式为：" #~ msgid "\\Pi_{\\mathcal{E}}: \\R^{Kd} \\to \\mathcal{E}: ( \\theta_1, \\ldots, \\theta_K) \\mapsto \\left(\\frac{1}{K}\\sum\\limits_{k=1}^K \\theta_K, \\ldots, \\frac{1}{K}\\sum\\limits_{k=1}^K \\theta_K \\right)." #~ msgstr "" @@ -400,5 +474,11 @@ msgstr "参考文献" #~ msgid "Psuedocode for ``Scaffold``" #~ msgstr "``Scaffold`` 伪代码" -#~ msgid "Variance reduction is a technique that can be flexibly combined with most algorithms and has been widely used in federated learning for dealing with statistical heterogeneity. However, it should be noted in the `SCAFFOLD algorithm `_ that on both the server and the clients, there are extra parameters :math:`c` and :math:`c_k` to maintain, which may increase the communication cost. In scenarios which are sensitive to communication cost, this would potentially be a problem. Therefore, a better solution could be a combination of the variance reduction technique and some **skipping** techniques (e.g. [:footcite:ct:`proxskip-vr`]), which will be introduced in next sections." -#~ msgstr "方差缩减是一种可以灵活地与大多数算法结合的技术，已在联邦学习中被广泛用于处理统计异质性。但要注意，在 `SCAFFOLD 算法 `_ 中，服务器与客户端上都有需要维护的额外参数 :math:`c` 与 :math:`c_k` ，这可能增加通信开销。在对通信开销敏感的场景中，这可能会成为问题。因此，更好的方案可能是把方差缩减技术与某种 **跳步** 技巧相结合（如 [:footcite:ct:`proxskip-vr`]），这将在后面几节介绍。" +#~ msgid "" +#~ "Variance reduction is a technique that can be flexibly combined with most algorithms and has been widely used in federated learning for dealing with statistical heterogeneity. However, it should be " +#~ "noted in the `SCAFFOLD algorithm `_ that on both the server and the clients, there are extra parameters :math:`c` and :math:`c_k` to maintain, which may increase the " +#~ "communication cost. In scenarios which are sensitive to communication cost, this would potentially be a problem. Therefore, a better solution could be a combination of the variance reduction " +#~ "technique and some **skipping** techniques (e.g. [:footcite:ct:`proxskip-vr`]), which will be introduced in next sections." +#~ msgstr "" +#~ "方差缩减是一种可以灵活地与大多数算法结合的技术，已在联邦学习中被广泛用于处理统计异质性。但要注意，在 `SCAFFOLD 算法 `_ 中，服务器与客户端上都有需要维护的额外参数 :math:`c` 与 :math:`c_k` ，这可能增加通信开销。在对通信开销敏感的场景中，这可能会成为问题。因此，更好的方案可能是把方差缩减技术与某种 **跳步** " +#~ "技巧相结合（如 [:footcite:ct:`proxskip-vr`]），这将在后面几节介绍。" diff --git a/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/proximal.po b/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/proximal.po index a0fe89fe..f3e8af4d 100644 --- a/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/proximal.po +++ b/docs/source/locale/zh_CN/LC_MESSAGES/algorithms/proximal.po @@ -24,8 +24,13 @@ msgid "Proximal Algorithms in Federated Learning" msgstr "联邦学习中的临近算法" #: ../../source/algorithms/proximal.rst:6 -msgid "In non-I.I.D. scenarios, based on the idea of reducing the impact of local updates of clients on the global model, [:footcite:ct:`sahu2018fedprox`] first introduced a proximal term to the local objective functions, aiming at making the algorithm more stable and converging faster. Compared to ``SCAFFOLD``, methods using proximal terms do not need to maintain extra parameters (mainly related to the gradients), hence having no communication overhead and no additional cost to security (refer to [:footcite:ct:`zhu2019deep_leakage`] for more details)." -msgstr "基于减小数据非独立同分布情形下、每一轮的子节点更新对整体模型影响的考虑，[:footcite:ct:`sahu2018fedprox`] 首先提出了在子节点上的局部模型中增加邻近项的联邦临近算法 ``FedProx`` ，达到了算法收敛更快、更加稳定的目的。相较于 ``SCAFFOLD`` 算法，使用邻近项的方案不需要进行额外（关于梯度信息）参数的维护与传递，没有通信开销以及安全性上的额外代价（详见 [:footcite:ct:`zhu2019deep_leakage`]）。" +msgid "" +"In non-I.I.D. scenarios, based on the idea of reducing the impact of local updates of clients on the global model, [:footcite:ct:`sahu2018fedprox`] first introduced a proximal term to the local " +"objective functions, aiming at making the algorithm more stable and converging faster. Compared to ``SCAFFOLD``, methods using proximal terms do not need to maintain extra parameters (mainly related" +" to the gradients), hence having no communication overhead and no additional cost to security (refer to [:footcite:ct:`zhu2019deep_leakage`] for more details)." +msgstr "" +"基于减小数据非独立同分布情形下、每一轮的子节点更新对整体模型影响的考虑，[:footcite:ct:`sahu2018fedprox`] 首先提出了在子节点上的局部模型中增加邻近项的联邦临近算法 ``FedProx`` ，达到了算法收敛更快、更加稳定的目的。相较于 ``SCAFFOLD`` " +"算法，使用邻近项的方案不需要进行额外（关于梯度信息）参数的维护与传递，没有通信开销以及安全性上的额外代价（详见 [:footcite:ct:`zhu2019deep_leakage`]）。" #: ../../source/algorithms/proximal.rst:12 msgid "To be more specific, in the :math:`(t+1)`-th iteration, the local objective function of client :math:`k` changes from :math:`f_k(\\theta_k)` to the following form with a proximal term:" @@ -41,7 +46,9 @@ msgid "" msgstr "" #: ../../source/algorithms/proximal.rst:24 -msgid "where :math:`\\mu` is a penalty constant. It should be noticed that the proximal center :math:`\\theta^{(t)}` is the model parameter on the server node obtained in the previous iteration (the :math:`t`-th iteration). Indeed, the overall optimization problem can be modeled as the following constrained optimization problem" +msgid "" +"where :math:`\\mu` is a penalty constant. It should be noticed that the proximal center :math:`\\theta^{(t)}` is the model parameter on the server node obtained in the previous iteration (the " +":math:`t`-th iteration). Indeed, the overall optimization problem can be modeled as the following constrained optimization problem" msgstr "上式中的 :math:`\\mu` 被称作罚参数 (Penalty Constant)。值得注意的是，上式邻近项的中心为 :math:`\\theta^{(t)}` ，即上一轮迭代得到的全局模型。实际上，联邦临近算法 ``FedProx`` 整体的优化问题可以建模为如下的约束优化问题" #: ../../source/algorithms/proximal.rst:28 @@ -53,7 +60,9 @@ msgid "" msgstr "" #: ../../source/algorithms/proximal.rst:36 -msgid "For alternatives for the proximal center, studies were conducted in [:footcite:ct:`hanzely2020federated, li_2021_ditto`] which would be introduced later. Now, we summarize the pseudocode for ``FedProx`` as follows:" +msgid "" +"For alternatives for the proximal center, studies were conducted in [:footcite:ct:`hanzely2020federated, li_2021_ditto`] which would be introduced later. Now, we summarize the pseudocode for " +"``FedProx`` as follows:" msgstr "关于邻近项中心的其他选取，[:footcite:ct:`hanzely2020federated, li_2021_ditto`] 进行了研究，我们随后进行详细介绍。我们将联邦临近算法 ``FedProx`` 的伪代码总结如下：" #: ../../source/algorithms/proximal.rst:43 @@ -71,15 +80,21 @@ msgid "" msgstr "" #: ../../source/algorithms/proximal.rst:55 -msgid "where :math:`\\prox_{f_k, \\mu}` is the proximal operator [:footcite:ct:`Moreau_1965_prox`] of :math:`f_k` with respect to :math:`\\mu`. Let :math:`s = \\frac{1}{\\mu}`, since one has :math:`\\prox_{f_k, \\mu} = \\prox_{sf_k, 1}`, we also denote :math:`\\prox_{f_k, \\mu}` as :math:`\\prox_{sf_k}`. Corresponding function" -msgstr "其中的 :math:`\\prox_{f_k, \\mu}` 为临近算子 (Proximity Operator) [:footcite:ct:`Moreau_1965_prox`]。令 :math:`s = \\frac{1}{\\mu}` ，因为 :math:`\\prox_{f_k, \\mu} = \\prox_{sf_k, 1}` ，我们也把 :math:`\\prox_{f_k, \\mu}` 记为 :math:`\\prox_{sf_k}` 。相应的函数" +msgid "" +"where :math:`\\prox_{f_k, \\mu}` is the proximal operator [:footcite:ct:`Moreau_1965_prox`] of :math:`f_k` with respect to :math:`\\mu`. Let :math:`s = \\frac{1}{\\mu}`, since one has " +":math:`\\prox_{f_k, \\mu} = \\prox_{sf_k, 1}`, we also denote :math:`\\prox_{f_k, \\mu}` as :math:`\\prox_{sf_k}`. Corresponding function" +msgstr "" +"其中的 :math:`\\prox_{f_k, \\mu}` 为临近算子 (Proximity Operator) [:footcite:ct:`Moreau_1965_prox`]。令 :math:`s = \\frac{1}{\\mu}` ，因为 :math:`\\prox_{f_k, \\mu} = \\prox_{sf_k, 1}` ，我们也把 :math:`\\prox_{f_k, " +"\\mu}` 记为 :math:`\\prox_{sf_k}` 。相应的函数" #: ../../source/algorithms/proximal.rst:59 msgid "\\mathcal{M}_{sf_k} (\\theta^{(t)}) = \\mathcal{M}_{f_k, \\mu} (\\theta^{(t)}) := \\inf\\limits_{\\theta_k} \\left\\{ f_k(\\theta_k) + \\frac{\\mu}{2} \\lVert \\theta_k - \\theta^{(t)} \\rVert^2 \\right\\}" msgstr "" #: ../../source/algorithms/proximal.rst:64 -msgid "is called **Moreau envelope** or **Moreau-Yosida regularization** of :math:`f_k` with respect to :math:`\\mu`. Moreau envelope of a function :math:`f_k` has the following relationship [:footcite:ct:`Parikh_2014_pa`] with its proximal operator:" +msgid "" +"is called **Moreau envelope** or **Moreau-Yosida regularization** of :math:`f_k` with respect to :math:`\\mu`. Moreau envelope of a function :math:`f_k` has the following relationship " +"[:footcite:ct:`Parikh_2014_pa`] with its proximal operator:" msgstr "被称作 :math:`f_k` 的 Moreau 包络 (Moreau Envelope) 或者 Moreau-Yosida 正规化 (Moreau-Yosida Regularization)。函数 :math:`f_k` 的 Moreau 包络与它的邻近算子有如下关系 [:footcite:ct:`Parikh_2014_pa`]：" #: ../../source/algorithms/proximal.rst:67 @@ -95,8 +110,15 @@ msgid "For the convergence of ``FedProx`` in non-I.I.D. scenarios, [:footcite:ct msgstr "关于联邦临近算法 ``FedProx`` 在非凸情况下的收敛性，[:footcite:ct:`sahu2018fedprox`] 有如下的结论：" #: ../../source/algorithms/proximal.rst:80 -msgid "Assume that the objective functions on clients :math:`\\{f_k\\}_{k=1}^K` are non-convex, :math:`L`-smooth (definition see :eq:`l-smooth`), and there exists a constant :math:`L_- > 0` such that :math:`\\nabla^2 f_k \\succcurlyeq -L_- I_d`. Assume further that the functions :math:`\\{f_k\\}_{k=1}^K` satisfy the so-called bounded dissimilarity condition, i.e. for any :math:`\\varepsilon > 0`, there exists a constant :math:`B_{\\varepsilon} > 0` such that for any point :math:`\\theta` in the set :math:`\\mathcal{S}_{\\varepsilon}^c := \\{ \\theta ~|~ \\lVert \\nabla f(\\theta) \\rVert^2 > \\varepsilon\\}`, the following inequality holds" -msgstr "假设子节点的目标函数 :math:`\\{f_k\\}_{k=1}^K` 都是非凸、:math:`L`-光滑（定义见 :eq:`l-smooth` ）函数，并且存在常数 :math:`L_- > 0` ，使得 :math:`\\nabla^2 f_k \\succcurlyeq -L_- I_d` 。又假设 :math:`\\{f_k\\}_{k=1}^K` 满足差异有界条件 (Bounded Dissimilarity)，即对 :math:`\\varepsilon > 0` ，存在常数 :math:`B_{\\varepsilon} > 0` ，使得集合 :math:`\\mathcal{S}_{\\varepsilon}^c := \\{ \\theta ~|~ \\lVert \\nabla f(\\theta) \\rVert^2 > \\varepsilon\\}` 中的任何一点 :math:`\\theta` 都满足不等式" +msgid "" +"Assume that the objective functions on clients :math:`\\{f_k\\}_{k=1}^K` are non-convex, :math:`L`-smooth (definition see :eq:`l-smooth`), and there exists a constant :math:`L_- > 0` such that " +":math:`\\nabla^2 f_k \\succcurlyeq -L_- I_d`. Assume further that the functions :math:`\\{f_k\\}_{k=1}^K` satisfy the so-called bounded dissimilarity condition, i.e. for any :math:`\\varepsilon > " +"0`, there exists a constant :math:`B_{\\varepsilon} > 0` such that for any point :math:`\\theta` in the set :math:`\\mathcal{S}_{\\varepsilon}^c := \\{ \\theta ~|~ \\lVert \\nabla f(\\theta) " +"\\rVert^2 > \\varepsilon\\}`, the following inequality holds" +msgstr "" +"假设子节点的目标函数 :math:`\\{f_k\\}_{k=1}^K` 都是非凸、:math:`L`-光滑（定义见 :eq:`l-smooth` ）函数，并且存在常数 :math:`L_- > 0` ，使得 :math:`\\nabla^2 f_k \\succcurlyeq -L_- I_d` 。又假设 :math:`\\{f_k\\}_{k=1}^K` 满足差异有界条件 (Bounded" +" Dissimilarity)，即对 :math:`\\varepsilon > 0` ，存在常数 :math:`B_{\\varepsilon} > 0` ，使得集合 :math:`\\mathcal{S}_{\\varepsilon}^c := \\{ \\theta ~|~ \\lVert \\nabla f(\\theta) \\rVert^2 > \\varepsilon\\}` " +"中的任何一点 :math:`\\theta` 都满足不等式" #: ../../source/algorithms/proximal.rst:86 msgid "B(\\theta) := \\frac{\\expectation_k [\\lVert \\nabla f_k(\\theta) \\rVert^2]}{\\lVert \\nabla f(\\theta) \\rVert^2} \\leqslant B_{\\varepsilon}." @@ -107,44 +129,81 @@ msgid "Fix constants :math:`\\mu, \\gamma` satisfying" msgstr "选取常数 :math:`\\mu, \\gamma` 满足" #: ../../source/algorithms/proximal.rst:93 -msgid "\\rho := \\left( \\frac{1}{\\mu} - \\frac{\\gamma B}{\\mu} - \\frac{B(1+\\gamma)\\sqrt{2}}{\\bar{\\mu}\\sqrt{K}} - \\frac{LB(1+\\gamma)}{\\bar{\\mu}\\mu} - \\frac{LB^2(1+\\gamma)^2}{2\\bar{\\mu}^2} - \\frac{LB^2(1+\\gamma)^2}{\\bar{\\mu}^2 K} \\left( 2\\sqrt{2K} + 2 \\right) \\right) > 0," +msgid "" +"\\rho := \\left( \\frac{1}{\\mu} - \\frac{\\gamma B}{\\mu} - \\frac{B(1+\\gamma)\\sqrt{2}}{\\bar{\\mu}\\sqrt{K}} - \\frac{LB(1+\\gamma)}{\\bar{\\mu}\\mu} - \\frac{LB^2(1+\\gamma)^2}{2\\bar{\\mu}^2} " +"- \\frac{LB^2(1+\\gamma)^2}{\\bar{\\mu}^2 K} \\left( 2\\sqrt{2K} + 2 \\right) \\right) > 0," msgstr "" #: ../../source/algorithms/proximal.rst:98 -msgid "where :math:`\\bar{\\mu} = \\mu - L_- > 0`. Then, in the :math:`(t+1)`-th iteration of ``FedProx``, assuming that the global model :math:`\\theta^{(t)}` of the previous iteration is not the first-order stationary point of the global objective function :math:`f(\\theta)`, (i.e. :math:`\\theta^{(t)} \\in \\mathcal{S}_{\\varepsilon}^c`), the following decrease in the global objective function holds" -msgstr "其中 :math:`\\bar{\\mu} = \\mu - L_- > 0` 。那么在 ``FedProx`` 的第 :math:`(t+1)` 步，设前一步的全局模型参数 :math:`\\theta^{(t)}` 为全局目标函数 :math:`f` 的一阶非稳定点（即 :math:`\\theta^{(t)} \\in \\mathcal{S}_{\\varepsilon}^c` ），那么 :math:`f` 的值满足如下的下降关系" +msgid "" +"where :math:`\\bar{\\mu} = \\mu - L_- > 0`. Then, in the :math:`(t+1)`-th iteration of ``FedProx``, assuming that the global model :math:`\\theta^{(t)}` of the previous iteration is not the first-" +"order stationary point of the global objective function :math:`f(\\theta)`, (i.e. :math:`\\theta^{(t)} \\in \\mathcal{S}_{\\varepsilon}^c`), the following decrease in the global objective function " +"holds" +msgstr "" +"其中 :math:`\\bar{\\mu} = \\mu - L_- > 0` 。那么在 ``FedProx`` 的第 :math:`(t+1)` 步，设前一步的全局模型参数 :math:`\\theta^{(t)}` 为全局目标函数 :math:`f` 的一阶非稳定点（即 :math:`\\theta^{(t)} \\in \\mathcal{S}_{\\varepsilon}^c` " +"），那么 :math:`f` 的值满足如下的下降关系" #: ../../source/algorithms/proximal.rst:102 msgid "\\expectation\\nolimits_{\\mathcal{S}^{(t)}}[f(\\theta^{(t+1)})] \\leqslant f(\\theta^{(t)}) - \\rho \\lVert \\nabla f (\\theta^{(t)}) \\rVert^2." msgstr "" #: ../../source/algorithms/proximal.rst:111 -msgid "For the `convergence theorem `_ of ``FedProx``, we have the following observations: in a neighbourhood of some zero of :math:`\\lVert \\nabla f \\rVert`, if this zero is not cancelled by :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]`, i.e. this point is also a zero of :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]` with the same or higher multiplicity, then in the neighbourhood, :math:`B_{\\varepsilon}` goes rapidly to infinity as :math:`\\varepsilon \\to 0`, thus violating the condition :math:`\\rho > 0`. In this case, the inequality :eq:`fedprox_obj_decrease` becomes meaningless." -msgstr "关于 ``FedProx`` 的 `收敛性定理 `_，我们有如下的观察：在 :math:`\\lVert \\nabla f \\rVert` 的零点附近，如果这个零点没有被 :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]` 消除掉，即这个点也是 :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]` 的零点，且零点的阶数相同或更高，那么在这个点的邻域内，:math:`B_{\\varepsilon}` 会随着 :math:`\\varepsilon \\to 0` 而急速趋向于无穷，导致在 :math:`\\lVert \\nabla f \\rVert` 的零点附近，:math:`\\rho > 0` 的假设不再成立，那么定理中的不等式 :eq:`fedprox_obj_decrease` 就变得无意义。" +msgid "" +"For the `convergence theorem `_ of ``FedProx``, we have the following observations: in a neighbourhood of some zero of :math:`\\lVert \\nabla f \\rVert`, if this zero is not cancelled" +" by :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]`, i.e. this point is also a zero of :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]` with the same or higher multiplicity, then in the " +"neighbourhood, :math:`B_{\\varepsilon}` goes rapidly to infinity as :math:`\\varepsilon \\to 0`, thus violating the condition :math:`\\rho > 0`. In this case, the inequality " +":eq:`fedprox_obj_decrease` becomes meaningless." +msgstr "" +"关于 ``FedProx`` 的 `convergence theorem `_，我们有如下的观察：在 :math:`\\lVert \\nabla f \\rVert` 的零点附近，如果这个零点没有被 :math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]` 消除掉，即这个点也是 " +":math:`\\mathbb{E}_k[\\lVert \\nabla f_k \\rVert]` 的零点，且零点的阶数相同或更高，那么在这个点的邻域内，:math:`B_{\\varepsilon}` 会随着 :math:`\\varepsilon \\to 0` 而急速趋向于无穷，导致在 :math:`\\lVert \\nabla f \\rVert` " +"的零点附近，:math:`\\rho > 0` 的假设不再成立，那么定理中的不等式 :eq:`fedprox_obj_decrease` 就变得无意义。" #: ../../source/algorithms/proximal.rst:117 -msgid "When the data distribution across clients is identical (ideal case), then :math:`B_{\\varepsilon}` is constantly equal to 1, which would not have the problem mentioned above. This problem is the start point of a series of follow-up works [:footcite:ct:`pathak2020fedsplit,tran2021feddr`]." +msgid "" +"When the data distribution across clients is identical (ideal case), then :math:`B_{\\varepsilon}` is constantly equal to 1, which would not have the problem mentioned above. This problem is the " +"start point of a series of follow-up works [:footcite:ct:`pathak2020fedsplit,tran2021feddr`]." msgstr "当子节点之间的数据分布完全一致的时候（理想情况下），:math:`B_{\\varepsilon}` 恒为 1，就不会有这个问题。这个问题也是后续一系列文章 [:footcite:ct:`pathak2020fedsplit,tran2021feddr`] 进行改进的出发点。" #: ../../source/algorithms/proximal.rst:120 -msgid "The positive significance of the ``FedProx`` algorithm is that it first introduced the proximal point algorithms (PPA) in the field of federated learning, although which were only used for solving local optimization problems (or equivalently the inner loop problem) and the whole of the ``FedProx`` algorithm is not a PPA in strict sense. The ``FedProx`` algorithm provides not only a good framework for theoretical analysis, but also a good starting point for the design of new algorithms. A large proportion of the algorithms proposed later for personalized fedrated learning [:footcite:ct:`hanzely2020federated, acar2021feddyn, li_2021_ditto, t2020pfedme, li2021pfedmac`] rely on the proximal terms (or similar terms) as the main technical tool for personalization." -msgstr "联邦临近算法 ``FedProx`` 的积极意义在于，首次将临近点算法 (Proximal Point Algorithms, PPA) 引入联邦学习领域中，虽然临近点算法只是用做了（内循环问题的）子节点上的子问题的求解器，而 ``FedProx`` 算法整体严格意义上来看并不是临近点算法。临近点算法不仅仅在理论分析上可以提供一个好的框架，在算法设计上也能提供一个支撑。后续在个性化联邦学习方面的工作 [:footcite:ct:`hanzely2020federated, acar2021feddyn, li_2021_ditto, t2020pfedme, li2021pfedmac`] 都依赖于引入邻近项（或者类似的项）作为模型个性化的主要技术手段。" +msgid "" +"The positive significance of the ``FedProx`` algorithm is that it first introduced the proximal point algorithms (PPA) in the field of federated learning, although which were only used for solving " +"local optimization problems (or equivalently the inner loop problem) and the whole of the ``FedProx`` algorithm is not a PPA in strict sense. The ``FedProx`` algorithm provides not only a good " +"framework for theoretical analysis, but also a good starting point for the design of new algorithms. A large proportion of the algorithms proposed later for personalized fedrated learning " +"[:footcite:ct:`hanzely2020federated, acar2021feddyn, li_2021_ditto, t2020pfedme, li2021pfedmac`] rely on the proximal terms (or similar terms) as the main technical tool for personalization." +msgstr "" +"联邦临近算法 ``FedProx`` 的积极意义在于，首次将临近点算法 (Proximal Point Algorithms, PPA) 引入联邦学习领域中，虽然临近点算法只是用做了（内循环问题的）子节点上的子问题的求解器，而 ``FedProx`` " +"算法整体严格意义上来看并不是临近点算法。临近点算法不仅仅在理论分析上可以提供一个好的框架，在算法设计上也能提供一个支撑。后续在个性化联邦学习方面的工作 [:footcite:ct:`hanzely2020federated, acar2021feddyn, li_2021_ditto, t2020pfedme, li2021pfedmac`] " +"都依赖于引入邻近项（或者类似的项）作为模型个性化的主要技术手段。" #: ../../source/algorithms/proximal.rst:214 msgid "``FedDC``: Federated Learning with Local Drift Decoupling and Correction" msgstr "``FedDC`` ：基于局部漂移解耦与校正的联邦学习" #: ../../source/algorithms/proximal.rst:216 -msgid "The proximal term of ``FedProx`` pulls each local model towards the global one, tacitly treating the gap between them as a nuisance to be suppressed. Under statistical heterogeneity, however, a nonzero drift between the local and the global model is inherent rather than incidental: the local models *should* deviate from the global one to better fit their own data distributions. The federated learning algorithm with local drift decoupling and correction (``FedDC``, see `arXiv:2203.11751 `_) therefore *decouples* the local drift into an auxiliary variable :math:`h_i` maintained by client :math:`i`, which tracks the gap between its local model and the global one. The ideal restriction :math:`h_i = w - \\theta_i` is relaxed into a penalized term, and the local objective of client :math:`i` in round :math:`t` reads" -msgstr "``FedProx`` 的临近项将每个局部模型拉向全局模型，默认把二者之间的差距视为应当被抑制的干扰。然而在统计异质性的情形下，局部模型与全局模型之间非零的漂移是固有的、而非偶然的：局部模型理应偏离全局模型，以便更好地拟合自身的数据分布。因此，具有局部漂移解耦与校正的联邦学习算法（ ``FedDC`` ，见 `arXiv:2203.11751 `_）将局部漂移解耦到由客户端 :math:`i` 维护的辅助变量 :math:`h_i` 中，用以刻画其局部模型与全局模型之间的差距。理想的约束 :math:`h_i = w - \\theta_i` 被松弛为一个罚项，客户端 :math:`i` 在第 :math:`t` 轮的局部目标函数为" +msgid "" +"The proximal term of ``FedProx`` pulls each local model towards the global one, tacitly treating the gap between them as a nuisance to be suppressed. Under statistical heterogeneity, however, a " +"nonzero drift between the local and the global model is inherent rather than incidental: the local models *should* deviate from the global one to better fit their own data distributions. The " +"federated learning algorithm with local drift decoupling and correction (``FedDC``, see `arXiv:2203.11751 `_) therefore *decouples* the local drift into an " +"auxiliary variable :math:`h_i` maintained by client :math:`i`, which tracks the gap between its local model and the global one. The ideal restriction :math:`h_i = w - \\theta_i` is relaxed into a " +"penalized term, and the local objective of client :math:`i` in round :math:`t` reads" +msgstr "" +"``FedProx`` 的临近项将每个局部模型拉向全局模型，默认把二者之间的差距视为应当被抑制的干扰。然而在统计异质性的情形下，局部模型与全局模型之间非零的漂移是固有的、而非偶然的：局部模型理应偏离全局模型，以便更好地拟合自身的数据分布。因此，具有局部漂移解耦与校正的联邦学习算法（ ``FedDC`` ，见 `arXiv:2203.11751 " +"`_）将局部漂移解耦到由客户端 :math:`i` 维护的辅助变量 :math:`h_i` 中，用以刻画其局部模型与全局模型之间的差距。理想的约束 :math:`h_i = w - \\theta_i` 被松弛为一个罚项，客户端 :math:`i` 在第 :math:`t` 轮的局部目标函数为" #: ../../source/algorithms/proximal.rst:226 msgid "F(\\theta_i; h_i, D_i, w) = L_i(\\theta_i) + \\frac{\\alpha}{2} \\left\\lVert \\theta_i + h_i - w \\right\\rVert^2 + \\frac{1}{\\eta K} \\langle \\theta_i, g_i - g \\rangle," msgstr "" #: ../../source/algorithms/proximal.rst:231 -msgid "where :math:`L_i` is the empirical loss on :math:`D_i`, :math:`\\alpha > 0` weights the parameter correction term, :math:`\\eta` is the learning rate, :math:`K` is the number of local iterations per round, and :math:`g_i` (resp. :math:`g`) denotes the last local update value :math:`g_i = \\theta_i^{+} - \\theta_i` (resp. its sample-size weighted average :math:`g = \\sum_{i=1}^N \\frac{\\lvert D_i \\rvert}{\\lvert D \\rvert} g_i`); the inner-product term, adopted from ``SCAFFOLD`` [:footcite:ct:`karimireddy2020scaffold`], corrects the gradient drift. Note that the center of the quadratic term is :math:`w - h_i`, i.e. the proximal center of ``FedProx`` corrected by the drift." -msgstr "其中 :math:`L_i` 为 :math:`D_i` 上的经验损失，:math:`\\alpha > 0` 为参数校正项的权重，:math:`\\eta` 为学习率，:math:`K` 为每轮的本地迭代次数；:math:`g_i` （相应地，:math:`g` ）表示上一轮的本地更新量 :math:`g_i = \\theta_i^{+} - \\theta_i` （及其按样本量的加权平均 :math:`g = \\sum_{i=1}^N \\frac{\\lvert D_i \\rvert}{\\lvert D \\rvert} g_i` ）；内积项沿用自 ``SCAFFOLD`` [:footcite:ct:`karimireddy2020scaffold`]，用以校正梯度漂移。注意二次项的中心是 :math:`w - h_i` ，即被漂移校正后的 ``FedProx`` 临近中心。" +msgid "" +"where :math:`L_i` is the empirical loss on :math:`D_i`, :math:`\\alpha > 0` weights the parameter correction term, :math:`\\eta` is the learning rate, :math:`K` is the number of local iterations per" +" round, and :math:`g_i` (resp. :math:`g`) denotes the last local update value :math:`g_i = \\theta_i^{+} - \\theta_i` (resp. its sample-size weighted average :math:`g = \\sum_{i=1}^N \\frac{\\lvert " +"D_i \\rvert}{\\lvert D \\rvert} g_i`); the inner-product term, adopted from ``SCAFFOLD`` [:footcite:ct:`karimireddy2020scaffold`], corrects the gradient drift. Note that the center of the quadratic " +"term is :math:`w - h_i`, i.e. the proximal center of ``FedProx`` corrected by the drift." +msgstr "" +"其中 :math:`L_i` 为 :math:`D_i` 上的经验损失，:math:`\\alpha > 0` 为参数校正项的权重，:math:`\\eta` 为学习率，:math:`K` 为每轮的本地迭代次数；:math:`g_i` （相应地，:math:`g` ）表示上一轮的本地更新量 :math:`g_i = \\theta_i^{+} - \\theta_i` （及其按样本量的加权平均" +" :math:`g = \\sum_{i=1}^N \\frac{\\lvert D_i \\rvert}{\\lvert D \\rvert} g_i` ）；内积项沿用自 ``SCAFFOLD`` [:footcite:ct:`karimireddy2020scaffold`]，用以校正梯度漂移。注意二次项的中心是 :math:`w - h_i` ，即被漂移校正后的 ``FedProx`` " +"临近中心。" #: ../../source/algorithms/proximal.rst:240 msgid "Each round, client :math:`i` initializes :math:`\\theta_i = w` and performs :math:`K` local gradient steps on :math:`F`, which in closed form amounts to" diff --git a/docs/source/locale/zh_CN/LC_MESSAGES/api/optimizers.po b/docs/source/locale/zh_CN/LC_MESSAGES/api/optimizers.po index 3fabd917..6973b57a 100644 --- a/docs/source/locale/zh_CN/LC_MESSAGES/api/optimizers.po +++ b/docs/source/locale/zh_CN/LC_MESSAGES/api/optimizers.po @@ -25,12 +25,9 @@ msgstr "" #: fl_sim.optimizers:4 of msgid "" -"This module contains the optimizers (local solvers) used in federated " -"learning. Despite optimizers from :mod:`torch.optim` and " -":mod:`torch_optimizer`, we also provide some custom optimizers for federated" -" learning for solving for example" -msgstr "" -"本模块包含联邦学习中所使用的优化器（局部求解器）。尽管优化器通常是为临近优化问题或拉格朗日对偶问题等特定形式的问题设计的，它们同样可以用于其他算法。" +"This module contains the optimizers (local solvers) used in federated learning. Despite optimizers from :mod:`torch.optim` and :mod:`torch_optimizer`, we also provide some custom optimizers for " +"federated learning for solving for example" +msgstr "本模块包含联邦学习中所使用的优化器（局部求解器）。尽管 :mod:`torch.optim` 与 :mod:`torch_optimizer` 中的优化器通常是为临近优化问题或拉格朗日对偶问题等特定形式的问题设计的，它们同样可以用于其他算法。我们还提供了一些面向联邦学习的自定义优化器，用于解决例如" #: fl_sim.optimizers:8 of msgid "proximal optimization problem" @@ -41,9 +38,7 @@ msgid "lagrangian dual problem" msgstr "拉格朗日对偶问题" #: fl_sim.optimizers:23::1 of -msgid "" -":py:obj:`get_optimizer `\\ " -"\\(optimizer\\_name\\, params\\, config\\)" +msgid ":py:obj:`get_optimizer `\\ \\(optimizer\\_name\\, params\\, config\\)" msgstr "" #: fl_sim.optimizers:23::1 of @@ -51,9 +46,7 @@ msgid "Get optimizer by name." msgstr "按名称获取优化器。" #: fl_sim.optimizers:23::1 of -msgid "" -":py:obj:`register_optimizer `\\ " -"\\(\\[name\\, override\\]\\)" +msgid ":py:obj:`register_optimizer `\\ \\(\\[name\\, override\\]\\)" msgstr "" #: fl_sim.optimizers:23::1 of