Scenarios / Distribution voltage

AI for Real-Time Voltage Control in Distribution Grids

Reactive power control of rooftop PV inverters

Shared objective

Keep the voltage of every bus between 0.95 and 1.05 p.u. with little reactive power

Each inverter observes

The voltage at its own bus, its solar output and its reactive power

Each inverter decides

Its reactive power, every 3 minutes

How the feeder responds to a sunny day

06:00

Without control

Unity power factor

Bus voltages along the feeder

With AI control

Adaptive

Bus voltages along the feeder

Negative reactive power: the inverters absorb reactive power, which lowers the voltage along their lines.

How to train such an AI efficiently?

Why conventional multi-agent training struggles at scale

One agent's share of the shared reward

Illustration with equal contributions: the blue slice is what agent i's own action causes.

Noise in each agent's learning signal

With conventional multi-agent reinforcement learning, every agent learns from the same reward, in which its own effect shrinks as the system grows.

How our algorithm trains the inverters, step by step

With global information about the whole feeder, all loads, all solar outputs and the line impedances, a centralized solver can compute the reactive power of every inverter that keeps the voltages in band with the least reactive power. Could such knowledge be internalized into how the agents are trained?

Referenceoffline, centralized

\tilde{q} (t) = \underset{q}{\arg \min} \frac{1}{B} \sum_{b} β (v_{b}) + \frac{0.1}{N} \sum_{i} | q_{i} | s.t. v = v^{0} (t) + X q, | q_{i} | \leq q_{i}^{\max} (t)

Reactive power of every inverter for the current loads and solar outputs: β penalizes voltages outside the band, v⁰ are the voltages without control, and X is the line reactance shared by two buses on their way to the substation (linearized power flow).

Guidanceper inverter

g_{i} (t) = \frac{1}{2} {(\frac{q_{i} (t) - {\tilde{q}}_{i} (t)}{s_{i}})}^{2} - \frac{1}{2} {(\frac{q_{i} (t - 1) - {\tilde{q}}_{i} (t)}{s_{i}})}^{2}

Change in the distance between inverter i's reactive power and its reference, scaled by its rating s_i; negative when it moves closer.

Learning signalper inverter

{\hat{A}}_{i} (t) = (1 - α) A (t) - α {\hat{g}}_{i} (t)

Shared advantage A of the reward r = −[mean β(v_b) + 0.1 × mean |q_i|], combined with the normalized guidance; α = 0.9.

Policy updateinternalizes the reference

\nabla_{θ} J \approx \sum_{t} \sum_{i} {\hat{A}}_{i} (t) \nabla_{θ} \log π_{θ} (a_{i} (t) | o_{i} (t))

Each inverter acts on its own observation o_i; the reference and the guidance are not needed at run time.

At its core, our algorithm uses prior knowledge of the system, in the form of a reference state, to guide the training of decentralized policies and give every agent a clear optimization signal. Through training, each inverter will internalize in its policy how to adjust its reactive power to what it observes at its own bus.

The charts below show A(t) and −ĝ_i(t) during one training run.

One inverter during training

Time

All inverters at the selected time

Learning progress

Highest voltage on the replayed day

Training days