Scenarios / Power generation

AI for Real-Time Distributed Power Generation Control

Real-time economic dispatch with frequency regulation

Shared objective

Keep the frequency at 50 Hz and minimize the total generation cost

Each generator observes

Grid frequency, load and generator outputs

Each generator decides

The change of its own output, every 2 seconds

How the grid responds to changing wind

00:00

Without control

Governors only

With AI control

Adaptive

Load and generation without control

Load and generation with AI control

Extra cost: generation cost above the cheapest way to produce the same total output.

How to train such an AI efficiently?

Why conventional multi-agent training struggles at scale

One agent's share of the shared reward

Illustration with equal contributions: the blue slice is what agent i's own action causes.

Noise in each agent's learning signal

With conventional multi-agent reinforcement learning, every agent learns from the same reward, in which its own effect shrinks as the system grows.

How our algorithm trains the generators, step by step

With global information about the whole system, a centralized solver can compute the optimal load, one that keeps generation and demand balanced at the lowest cost. Could such knowledge be internalized into how the agents are trained?

Referenceoffline, centralized

\tilde{p} (t) = \underset{p}{\arg \min} \sum_{i} C_{i} (p_{i}) s.t. \sum_{i} p_{i} = P_{L} (t), p_{i}^{\min} \leq p_{i} \leq p_{i}^{\max}

Least-cost output of every generator for the current net load P_L.

Guidanceper generator

g_{i} (t) = \frac{1}{2} {(\frac{u_{i} (t + 1) - {\tilde{p}}_{i} (t)}{p_{i}^{\max}})}^{2} - \frac{1}{2} {(\frac{u_{i} (t) - {\tilde{p}}_{i} (t)}{p_{i}^{\max}})}^{2}

Change in the distance between generator i's command u_i and its reference output; negative when it moves closer.

Learning signalper generator

{\hat{A}}_{i} (t) = (1 - α) A (t) - α {\hat{g}}_{i} (t)

Shared advantage A of the reward r = −[(Δf / 0.05 Hz)² + extra cost], combined with the normalized guidance; α = 0.9.

Policy updateinternalizes the reference

\nabla_{θ} J \approx \sum_{t} \sum_{i} {\hat{A}}_{i} (t) \nabla_{θ} \log π_{θ} (a_{i} (t) | o_{i} (t))

Each generator acts on its own observation o_i; the reference and the guidance are not needed at run time.

At its core, our algorithm uses prior knowledge of the system, in the form of a reference state, to guide the training of decentralized policies and give every agent a clear optimization signal. Through training, each agent will internalize in its policy how to adjust its power output to what it observes.

The charts below show A(t) and −ĝ_i(t) during one training run.

One generator during training

Time

All generators at the selected time

One training run after 5 episodes. Separate vertical scales. Click a bar to select a generator.

Learning progress

Extra cost on the replayed disturbance

Episode