Energy demand profile: to take heating into account#28232
Conversation
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 4 issues, and left some high level feedback:
- Access to
loadpointEnergyandloadpointSlotStartmaps happens from multiple goroutines inupdateLoadpoints/updateLoadpointConsumptionwithout synchronization, which risks concurrent map access panics; consider guarding these maps (and their contained state) with a mutex or encapsulating per-loadpoint state in a concurrency-safe structure. - In
applyTemperatureCorrection, the loopfor h := range 24is invalid in Go and will not compile; it should be replaced with an index loop likefor h := 0; h < 24; h++when buildingpastTempAvg.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Access to `loadpointEnergy` and `loadpointSlotStart` maps happens from multiple goroutines in `updateLoadpoints`/`updateLoadpointConsumption` without synchronization, which risks concurrent map access panics; consider guarding these maps (and their contained state) with a mutex or encapsulating per-loadpoint state in a concurrency-safe structure.
- In `applyTemperatureCorrection`, the loop `for h := range 24` is invalid in Go and will not compile; it should be replaced with an index loop like `for h := 0; h < 24; h++` when building `pastTempAvg`.
## Individual Comments
### Comment 1
<location path="core/site_optimizer.go" line_range="658-659" />
<code_context>
+ pastTempCount[h]++
+ }
+ }
+ pastTempAvg := make([]float64, 24)
+ for h := range 24 {
+ if pastTempCount[h] > 0 {
+ pastTempAvg[h] = pastTempSum[h] / float64(pastTempCount[h])
</code_context>
<issue_to_address>
**issue (bug_risk):** The `for h := range 24` loop is invalid Go and will not compile.
Use a standard indexed `for` loop here instead of ranging over an `int`, for example:
```go
pastTempAvg := make([]float64, 24)
for h := 0; h < 24; h++ {
if pastTempCount[h] > 0 {
pastTempAvg[h] = pastTempSum[h] / float64(pastTempCount[h])
}
}
```
</issue_to_address>
### Comment 2
<location path="core/site.go" line_range="927-928" />
<code_context>
)
- for _, lp := range site.loadpoints {
+ for i, lp := range site.loadpoints {
+ lpID := i // capture loop variable for goroutine
wg.Go(func() {
power := lp.UpdateChargePowerAndCurrents()
</code_context>
<issue_to_address>
**issue (bug_risk):** The goroutine still closes over the loop variable `lp`, which can cause data races and incorrect behavior.
You correctly capture `lpID`, but `lp` is still shared across iterations due to `range` semantics. Capture `lp` in a new local variable before starting the goroutine:
```go
for i, lp := range site.loadpoints {
lpID := i
lp := lp // capture lp for goroutine
wg.Go(func() {
power := lp.UpdateChargePowerAndCurrents()
// ...
})
}
```
This prevents the goroutine from seeing a different loadpoint than intended and avoids subtle concurrency bugs in the update methods.
</issue_to_address>
### Comment 3
<location path="core/site_optimizer.go" line_range="687-688" />
<code_context>
+ continue
+ }
+
+ h := ts.UTC().Hour()
+ tPastAvg := pastTempAvg[h]
+
+ // delta > 0: tomorrow colder than historical average → load increases
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Using a zero default for `tPastAvg` when there is no historical data for a given hour can skew the correction significantly.
When `pastTempCount[h] == 0`, `pastTempAvg[h]` stays at its zero value, which makes `tFuture` look much warmer than "history" and can drive overly strong negative corrections for those hours.
Consider skipping correction or using a safer fallback (e.g. overall past 24h average) when there’s no data:
```go
h := ts.UTC().Hour()
if pastTempCount[h] == 0 {
continue // or use a fallback average
}
tPastAvg := pastTempAvg[h]
```
This prevents over-correcting when the historical baseline is unknown.
```suggestion
h := ts.UTC().Hour()
if pastTempCount[h] == 0 {
// no historical data for this hour; skip correction (or use a fallback if available)
continue
}
tPastAvg := pastTempAvg[h]
```
</issue_to_address>
### Comment 4
<location path="core/site_optimizer.go" line_range="515" />
<code_context>
func (site *Site) homeProfile(minLen int) ([]float64, error) {
- // kWh over last 30 days
- profile, err := metrics.Profile(now.BeginningOfDay().AddDate(0, 0, -30))
+ from := now.BeginningOfDay().AddDate(0, 0, -7)
+
+ // kWh average over last 7 days - base load (excludes loadpoints)
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring the new home profile and temperature-correction logic into small reusable helpers and pre-indexed data structures to keep the core control flow simple and readable.
The new logic is valid but it does increase complexity in a few focused places. You can reduce it without changing behavior by extracting small helpers and pre-indexing data.
### 1) Repeated “repeat profile until minLen then trim” logic
This pattern appears multiple times in `homeProfile` for both base and heater profiles:
```go
slots := make([]float64, 0, minLen+1)
for len(slots) <= minLen+24*4 { // allow for prorating first day
slots = append(slots, gt_base[:]...)
}
res := profileSlotsFromNow(slots)
if len(res) < minLen {
return nil, fmt.Errorf("minimum home profile length %d is less than required %d", len(res), minLen)
}
if len(res) > minLen {
res = res[:minLen]
}
```
and similarly for `heaterSlots`.
This can be encapsulated in a reusable helper:
```go
func repeatAndTrimProfile(profile []float64, minLen int) ([]float64, error) {
slots := make([]float64, 0, minLen+1)
for len(slots) <= minLen+24*4 {
slots = append(slots, profile...)
}
res := profileSlotsFromNow(slots)
if len(res) < minLen {
return nil, fmt.Errorf("minimum home profile length %d is less than required %d", len(res), minLen)
}
if len(res) > minLen {
res = res[:minLen]
}
return res, nil
}
```
Then `homeProfile` becomes simpler:
```go
gtBaseSlots, err := repeatAndTrimProfile(gt_base, minLen)
if err != nil {
return nil, err
}
var gtHeaterSlots []float64
if len(gt_heater_raw) > 0 {
if heater, err := repeatAndTrimProfile(gt_heater_raw, len(gtBaseSlots)); err == nil {
gtHeaterSlots = heater
}
}
```
This removes duplicated branching and makes the length-normalization logic easier to reason about.
### 2) Split `homeProfile` into clearer orchestration steps
`homeProfile` is currently doing: read base, read heater, normalize both, correct heater by temperature, merge, convert units. Extracting those as small helpers keeps the control flow readable:
```go
func (site *Site) buildBaseProfile(from time.Time, minLen int) ([]float64, error) {
base, err := metrics.Profile(from)
if err != nil {
return nil, err
}
return repeatAndTrimProfile(base, minLen)
}
func (site *Site) buildHeaterProfile(from time.Time, minLen int) []float64 {
raw := site.extractHeaterProfile(from, time.Now())
if len(raw) == 0 {
return nil
}
heater, err := repeatAndTrimProfile(raw, minLen)
if err != nil {
return nil
}
return site.applyTemperatureCorrection(heater)
}
```
`homeProfile` then reduces to an orchestrator:
```go
func (site *Site) homeProfile(minLen int) ([]float64, error) {
from := now.BeginningOfDay().AddDate(0, 0, -7)
base, err := site.buildBaseProfile(from, minLen)
if err != nil {
return nil, err
}
heater := site.buildHeaterProfile(from, len(base))
final := make([]float64, len(base))
for i := range base {
final[i] = base[i]
if heater != nil && i < len(heater) {
final[i] += heater[i]
}
}
return lo.Map(final, func(v float64, _ int) float64 { return v * 1e3 }), nil
}
```
Same behavior, but the main function reads as a high-level description of the steps.
### 3) Avoid O(N²) search in `applyTemperatureCorrection`
The inner loop that finds `tFuture` completely dominates the function’s complexity and obscures the main idea:
```go
for i := range profile {
ts := slotStart.Add(time.Duration(i) * tariff.SlotDuration)
var tFuture float64
found := false
for _, r := range rates {
if r.Start.Equal(ts) {
tFuture = r.Value
found = true
break
}
}
if !found {
continue
}
// ...
}
```
You can pre-index the forecast into a map and keep the slot loop single-level:
```go
func indexRatesByStart(rates api.Rates) map[time.Time]float64 {
m := make(map[time.Time]float64, len(rates))
for _, r := range rates {
m[r.Start] = r.Value
}
return m
}
```
Use it in `applyTemperatureCorrection`:
```go
forecastByTime := indexRatesByStart(rates)
slotStart := currentTime.Truncate(tariff.SlotDuration)
for i := range profile {
ts := slotStart.Add(time.Duration(i) * tariff.SlotDuration)
tFuture, ok := forecastByTime[ts]
if !ok {
continue
}
h := ts.UTC().Hour()
tPastAvg := pastTempAvg[h]
delta := tPastAvg - tFuture
result[i] = profile[i] * (1 + coeff*delta)
}
```
This both improves performance and reduces nesting, making the correction logic easier to follow.
### 4) Separate guard/validation from the core correction
The top of `applyTemperatureCorrection` is mostly guards and configuration checks. Extracting them into a small helper can clarify the “happy path”:
```go
func (site *Site) getWeatherRates() (api.Rates, float64, float64, error) {
weatherTariff := site.GetTariff(api.TariffUsageTemperature)
if weatherTariff == nil {
return nil, 0, 0, fmt.Errorf("no weather tariff")
}
rates, err := weatherTariff.Rates()
if err != nil || len(rates) == 0 {
return nil, 0, 0, fmt.Errorf("no weather rates")
}
threshold := site.HeatingThreshold
coeff := site.HeatingCoefficient
if threshold == 0 || coeff == 0 {
return nil, 0, 0, fmt.Errorf("heating config missing")
}
return rates, threshold, coeff, nil
}
```
Then:
```go
func (site *Site) applyTemperatureCorrection(profile []float64) []float64 {
rates, threshold, coeff, err := site.getWeatherRates()
if err != nil {
return profile
}
// ... 24h avg, hourly history, forecastByTime, main loop ...
}
```
This keeps `applyTemperatureCorrection` focused on the actual correction math rather than the early-exit conditions.
These refactors maintain all functionality but reduce branching and nesting, making the new behavior easier to understand and maintain.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
First thing is to get #23185 in. |
|
This contribution does not appear to meet our AI contribution guidelines. |
- Fix goroutine loop variable capture for loadpoint - Skip temperature correction when no historical data for hour - Add debug logging for missing historical temperature data Addresses review comments from PR evcc-io#28232
|
@naltatis this PR shows general understanding of evcc and has prior PRs. Fine for me. |
|
ill add more evidence for local testing and benefits once I have the build green. should be there is a bit. ai reviews addressed as follows |
|
ok, ready for review. because Base PR: #27780 (Weather Tariff) is open, only look at these 3 files below. The rest are the temperature tariff code changes that go away once that PR is merged. 
ill continue local testing and add evidence from the sqllite db, about migration (using my evcc.db file from latest) and most importantly, about the added precision of household load forecasts. With this PR optimizer forecasts are finally useable when you have a heatpump device registered in evcc. Before that they were unuseable because:
|
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In applyTemperatureCorrection, the loop
for h := range 24will not compile in Go; it should be replaced with a standard indexed loop such asfor h := 0; h < 24; h++ { ... }. - The temperature correction currently relies on an exact
time.Timematch betweenslotStart.Add(i*SlotDuration)andratesByTimekeys; to avoid missed corrections due to timezone or truncation mismatches, consider deriving future slots directly from the tariff rates or using a nearest-slot lookup instead of exact equality.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In applyTemperatureCorrection, the loop `for h := range 24` will not compile in Go; it should be replaced with a standard indexed loop such as `for h := 0; h < 24; h++ { ... }`.
- The temperature correction currently relies on an exact `time.Time` match between `slotStart.Add(i*SlotDuration)` and `ratesByTime` keys; to avoid missed corrections due to timezone or truncation mismatches, consider deriving future slots directly from the tariff rates or using a nearest-slot lookup instead of exact equality.
## Individual Comments
### Comment 1
<location path="core/site_optimizer.go" line_range="659-660" />
<code_context>
+ pastTempCount[h]++
+ }
+ }
+ pastTempAvg := make([]float64, 24)
+ for h := range 24 {
+ if pastTempCount[h] > 0 {
+ pastTempAvg[h] = pastTempSum[h] / float64(pastTempCount[h])
</code_context>
<issue_to_address>
**issue (bug_risk):** The `for h := range 24` loop does not compile and should iterate over a slice or a numeric range explicitly.
`range` requires an array, slice, map, string, or channel. Here you probably want either `for h := range pastTempAvg { ... }` or `for h := 0; h < 24; h++ { ... }`. Given the fixed size, iterating over `pastTempAvg` is likely the most idiomatic option.
</issue_to_address>
### Comment 2
<location path="core/site.go" line_range="96-97" />
<code_context>
householdEnergy *meterEnergy
householdSlotStart time.Time
+ // per-loadpoint energy tracking for heating devices
+ loadpointEnergy map[int]*meterEnergy
+ loadpointSlotStart map[int]time.Time
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Concurrent access to `loadpointEnergy` and `loadpointSlotStart` maps is not synchronized and can cause data races.
These maps are written in `updateLoadpointConsumption`, which runs in goroutines spawned by `updateLoadpoints`. Even if goroutines touch different keys, Go maps are not safe for concurrent writes and can panic (`concurrent map writes`) or introduce data races. Please either switch to a slice indexed by loadpoint ID or protect map access with synchronization (e.g., a shared mutex or a per-loadpoint struct with an embedded mutex).
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
update: db schema changes removed. see PR description. no more additional loadpoint column. 
|
andig
changed the title
|
@andig I removed any db schema change from this PR, so this is now independent of #23185. see Just tested compatibility with v0.303. works fine without issues. also, table "meter" remains lean which is a good thing. adding a varchar added significant volume to the file (20 bytes per each row...) the offset between meter ids and loadpoint ids is now 1000 to give enough space between the two for all practical numbers of meters and loadpoints. |
a5764b2 to
16203c8
Compare
|
evidence of the entire process working and about the impact on optimizer forecast and corrections. log messages of a working system: -> see how the heater profile slots got adjusted based on temperature forecast here is the result in the optimizer. see the wavy pattern of a modulating heatpump, how de-icing was running around 5am and how warm water was produced starting around noon. 
now here is the same optimizer profile without the temperature adjustments (you see differences in the range of 10% as per log messages above temperature didnt change that much between profile avg and tomorrow)
and finally, here is the exact same optimizer picture from vanilla evcc v0.303. Completely different household load and honestly not very useable/realistic. 
####################### @andig relevant code is in these 3 files. the remaining changes are all from the base PR (temperature tariff) 
|
Adds optional temperature correction to household load forecasts for the optimizer, allowing more accurate predictions when heating/cooling loads vary with outdoor temperature. Configuration (optional, in site config): - heatingThreshold: °C 24h avg above which corrections are disabled (default 12°C) - heatingCoefficient: fractional load change per °C delta (default 0.05 = 5%) Algorithm: 1. Gates on past 24h avg temperature vs threshold (heating active if below) 2. For each future slot, compares forecast temp to historical avg at same hour 3. Adjusts load: load *= (1 + coeff * (T_historical_avg - T_forecast)) - Colder forecast → higher load estimate - Warmer forecast → lower load estimate Requires the Temperature tariff (TariffUsageTemperature) to be configured. Changes household profile lookback from 30 days to 7 days because: - 7 days follows household rhythms more closely (weekly patterns) - Temperature changes too much over 30 days, making older data less relevant
- Separate heater load from base household consumption - Apply temperature correction only to heating devices (heat pumps, electric heaters) - Base loads (lighting, appliances) remain unchanged - Backward compatible: falls back to old behavior if no heating devices Changes: - Extended metrics DB to track per-loadpoint consumption - Added loadpoint energy tracking infrastructure in Site - Implemented profile extraction and aggregation functions - Modified homeProfile() to separate, correct, and merge profiles Addresses feedback on PR evcc-io#27780 that temperature adjustment should only apply to heating devices, not entire household consumption.
…aniel309/evcc into feature/temperature-correction
@daniel309 coming from evcc-io/optimizer#85 (comment): I understand this PR covers different classes of heating scenarios- temperature dependent (room heating) and schedule dependent. We already have schedule dependent in terms of averaged daily profile. Does this PR add more capabilities to that (week profile rather than day profile)? It seems to me that we should further split this PR into
Does that make sense? |
yes, this PR adds a weekly profile for heating. and then does outdoor temperature-dependent scaling (based on the weather tariff PR) of each slot's power usage based on a model that simulates house insulation.
yes, splitting into daily, weekly and temperature dependant (i.e. scaling power on top of daily or weekly profile) is exactly whats needed imo. |
|
...and that would include a mechanism for selecting the demand profile strategy |
Yes. This is why I added the
flag in this PR. we would probably need 3 features on top of "heating" or to replace "heating":
|
|
No ;). Heating is just that- changes the loadpoint appearance. I think these are new DemandProfileXYZ features. Orthogonal and giving much more flexibility? |
right. And it would allow combination, e.g. a loadpoint is a "heating", and has demand profile "weekly"/room heater and "scalebyoutdoortemp" (heatpump or AC for room heating). whereas another one is heating, and only of demandprofile daily/warmwater (heating rod). |
…ekly/Temperature Introduce two orthogonal demand profile feature flags for heating devices: - DemandProfileWeekly: uses same-weekday-last-week actual load (warm water) - DemandProfileTemperature: uses daily-avg load scaled by outdoor temp (room heating) Adds energyProfileWeekday() DB query and Collector.EnergyProfileWeekday() method. Extracts tileAndTrim() helper. Removes getHeatingLoadpoints(). Addresses andig feedback on PR evcc-io#28232.
|
@andig I merged all changes to latest master, now the diff is much cleaner (only 5 files changed). Also, I implemented the two demand profiles in the last commit: DemandProfileWeekly — for warm water heaters (boilers, heating rods) DemandProfileTemperature — for room heating heat pumps or any other electrical room heating system (AC heaters, IR heaters, ...) |
| // base load (excludes loadpoints) - averaged over last 30 days | ||
| gt_base, err := site.collectors[metrics.Home].EnergyProfile(now.BeginningOfDay().AddDate(0, 0, -30)) | ||
| // base load (excludes loadpoints) - averaged over last 7 days | ||
| gt_base, err := site.collectors[metrics.Home].EnergyProfile(now.BeginningOfDay().AddDate(0, 0, -7)) |
There was a problem hiding this comment.
I don‘t think we‘d want to do that
There was a problem hiding this comment.
right, this is household base load. avg across 30 days is probably fine here. I dont see periodic behavior that would be hidden/removed by averaging.
ill fix in next commit
There was a problem hiding this comment.
I thought this PR added detecting weekly patterns (DemandProfileWeekly)- those would be hidden?
There was a problem hiding this comment.
I thought this PR added detecting weekly patterns (DemandProfileWeekly)
yes, but not for the base load, only for heating "loadpoints".
That sounds to me like an argument for averaging over each weekday, not for taking only the last week as authorative? |
no ;-) if you average over each weekday you lose the weekly periodic behaviour. If you want to average, you can create same slot avg for the same weekday, e.g. take slot 12:00 to 12:15 of the past 4 fridays and avg load across those 4 slots as the new forecast for 12:00 to 12:15 of the coming friday. ... and yes, as discussed in the other PR: a proper time-series prediction model (tiny time mixers, googles TimesFM or more traditional methods like ARIMA) is in order longer term for the This completely differs from |
8 participants
Temperature-Based Household Load Correction with Heater Profile Separation
Base PR: #27780 (Weather Tariff)
This PR: Heater Profile Separation for Temperature Correction
note: this PR doesn't change the database schema. It is based on, and uses the "history for meters" work merged via #23185.
TL/DR
With this PR optimizer forecasts for the household load are significantly improved when you have a heatpump device registered in evcc.
Before that they were barely useable because:
evidence of the effects of this PR: #28232 (comment)
explanation of the mathematical function to estimate corrections: #28232 (comment)
Problem
The evopt energy optimizer uses a household load profile (
gt) to predict how much energy the home will consume in each future 15-minute slot. This profile is currently computed as a 30-day historical average — the same flat pattern is repeated regardless of weather conditions.This is a significant blind spot: heating and cooling loads are strongly temperature-dependent. On a cold winter day, a home with a heat pump or electric heating can consume 30–80% more energy than on a mild day. When the optimizer doesn't know this, it:
Important: The evcc optimizer is a mathematical solver that optimizes EV charging and battery schedules based on given inputs (solar forecast, grid prices, household load). It cannot forecast household load itself—it requires accurate future household load predictions as input. Without temperature-corrected heat pump load forecasts, the optimizer receives inaccurate household demand predictions, leading to suboptimal charging decisions that conflict with actual heating needs.
Solution Overview
This PR builds on #27780 (which added the
open-meteoweather tariff) and implements temperature-based correction of the household load profile with a critical improvement: the correction is applied only to heating device loads that are explicitly marked as temperature-sensitive.Key Innovation: Selective Temperature Correction
Not all heating devices have temperature-dependent loads. For example:
This PR implements a four-step process:
OutdoorTemperatureSensitivefeature flagNote on Historical Data Periods:
Temperature Correction Algorithm
The correction algorithm uses a physics-based model that relates heating load to the temperature difference between indoor and outdoor conditions:
which becomes:
where:
T_room= 21°C (constant room temperature)T_past_avg[h]= average temperature at hour-of-dayhover the past 7 daysT_forecast[i]= forecast temperature at the wall-clock time of slotiThis formula models heating load as proportional to the temperature difference that must be maintained. The correction factor represents the ratio of future heating demand to historical average demand.
Safety Check: If historical temperature is within 0.5°C of room temperature (indicating heating was likely off), the correction is skipped for that time slot.
Clamping: Correction factors are clamped to the range [0.5, 2.0] to prevent extreme corrections from bad data.
Example: With a 7-day historical average of 8°C at a given hour:
Configuration
The feature is fully opt-in — no changes needed for existing setups. Temperature correction is only active for heating devices explicitly marked as temperature-sensitive.
Basic Setup (requires base PR #27780)
Configuration Details
Temperature Tariff (required for correction):
Heating Device Features:
heating- Marks the device as a heating system (required for all heating devices)outdoortemperaturesensitive- Enables temperature correction for this specific device (optional)Important:
heatingfeature will have their consumption included in forecasts but without temperature correctionheatingandoutdoortemperaturesensitivewill have temperature correction appliedBackward Compatibility
The feature is completely opt-in and requires explicit configuration. Existing setups are unaffected.
Multiple Heater Support
The implementation fully supports multiple heating devices with selective correction:
api.Heatingfeature are automatically identifiedapi.OutdoorTemperatureSensitiveget temperature correctionThis approach ensures that:
Database Schema
Uses the new entity and meters tables introduced by PR #23185.
Dependencies
api.Heatingfeature flag for device identificationapi.OutdoorTemperatureSensitivefeature flag for selective correction