Physics-Informed Transformer for Rapid In-Situ U-Value Estimation

Notebook-free research and MLOps codebase for rapid building-envelope U-value estimation from short-duration transient heat-flux-meter style measurements.

The implementation follows the paper abstract directly: synthetic pretraining, dynamic RC-wall thermal simulation, ISO/LSTM/Transformer comparisons, physics-informed Transformer loss terms, duration studies, MLflow tracking, and uncertainty-aware prediction intervals.

Completed TCN Manuscript Benchmark

The current manuscript is paper/engineering_proceedings/manuscript.tex. Its fixed benchmark configuration and public evidence manifest are:

• configs/site_adaptation_all_public_tcn_transfer_48h_72h_mps_10k.yaml
• reports/all_public_tcn_transfer_48h_72h_mps_10k_exploratory/README.md

The protocol uses strictly synthetic-only pretraining and normalization, followed by target-site calibration. On the four normal-gradient records, pretraining reduced mean q RMSE versus the site-only TCN by 8.4% at 48 h and 9.2% at 72 h, with 12/12 and 11/12 paired wins. This bounded result does not establish improved integrated U or replacement of HFM measurements or ISO 9869-1.

After preparing DOI-sourced public data and regenerating the untracked synthetic corpus, verify the fixed profile without launching training:

uv run python scripts/run_all_public_tcn_transfer.py --preflight

The complete locked-environment and reproduction commands are in the report manifest. The exact executed profile retained its original exploratory label and was retrospectively promoted to manuscript evidence.

Release State

Current release: v0.18.2-site-trace-48h

Implemented:

• Two-node transient RC-wall simulator backed by explicit multilayer wall assemblies with randomized U-value, capacitance, wall archetype, weather profile, solar disturbance, sensor noise, heat-flux bias, and drift.
• Synthetic pretraining corpus config in configs/synthetic.yaml.
• Real HFM validation and retraining path via scripts/validate_real_data.py, scripts/retrain_real.py, and configs/real_retrain.yaml.
• Public DOI-indexed HFM data workflow via scripts/download_public_hfm.py, scripts/prepare_public_hfm.py, and configs/public_real_retrain.yaml.
• Separate MDPI-style heat-flux forecasting protocol via scripts/run_mdpi_protocol.py and configs/mdpi_protocol.yaml, with per-series train/validation/test separation.
• Ten-seed public-HFM MDPI benchmark via scripts/run_mdpi_seed_grid.py and configs/mdpi_protocol_10seed.yaml.
• Synthetic-to-public transfer protocol via scripts/run_mdpi_transfer.py and configs/mdpi_transfer.yaml: synthetic train/validation/test pretraining, public per-series train/validation/test fine-tuning, and live MLflow epoch tracking.
• Extra-long Transformer transfer diagnostic via configs/mdpi_transfer_transformers_xlong.yaml, with 600 synthetic pretrain epochs, 900 public fine-tune epochs, CUDA execution, MLflow curves, and no final train+validation refit.
• Real-world transfer protocol via configs/mdpi_transfer_realworld.yaml: quality-gated U deployment, train-derived RC prior feature for PhysicsTransformer, robust Huber public fine-tuning, HFM-vs-reference target-stability reporting, and an optional validation-selected selector diagnostic that is disabled by default.
• Strict q-to-U transfer config via configs/mdpi_transfer_q_to_u_strict.yaml: neural models estimate heat flux first, and reported U is computed from q_hat by ISO-style averaging.
• Strict q-to-U Optuna/Hydra search via configs/mdpi_transfer_hpo_q_to_u.yaml, with fixed lambda_u: 0.0, physics_u_estimator: flux, and robust upper-trimmed validation aggregation.
• Calibrated residual q-to-U HPO via configs/mdpi_transfer_hpo_q_to_u_calibrated.yaml: train-derived ISO heat-flux prior, zero-initialized residual public fine-tuning, validation residual shrinkage, within-family Optuna search, and 10-seed final public-HFM evaluation.
• Conservative ISO/LSTM deployment gate via configs/mdpi_transfer_q_to_u_lstm_conservative_selector.yaml: keep ISO unless LSTM validation q RMSE is less than 0.6x the ISO validation q RMSE.
• Final U-value comparison analysis via scripts/analyze_final_u_values.py: reads the best calibrated HPO final metrics, adds the conservative ISO/LSTM gate, compares per-seed versus seed-averaged U estimates, and writes final U-value graphs to reports/final_u_value_comparison/.
• Final q-prediction diagnostics via scripts/run_final_q_prediction_diagnostics.py: regenerates q predictions from selected final HPO hyperparameters and writes synthetic/public q_pred versus q_true plots to reports/final_q_prediction_diagnostics/.
• Deployable calibrated q-to-U bundle path via scripts/train_q_to_u_bundle.py, scripts/evaluate_q_to_u_bundle.py, and src/uvalue/deployment/q_to_u_bundle.py.
• Trusted-path ONNX q-forecaster export via scripts/export_q_to_u_onnx.py, with metadata describing normalized feature inputs and q_pred output.
• FastAPI q-to-U inference endpoint via POST /estimate-u, including calibration rows, optional validation rows, conservative ISO fallback decision, and Prometheus counters.
• Local MLOps smoke pipeline via scripts/run_local_mlops_pipeline.py: train bundle, evaluate bundle, export ONNX, start service, send /estimate-u, and write a report.
• U-aware PhysicsTransformer transfer HPO via configs/mdpi_transfer_hpo_uaware_physics.yaml: direct synthetic U-head supervision, validation-calibrated flux/direct U blending, Optuna search, 10-seed final evaluation, and a documented rejected-ablation diagnostic.
• Conference-oriented windowed sim-to-real runner via scripts/run_conference_sim_to_real.py and configs/conference_sim_to_real_q_to_u.yaml: synthetic pretraining, public-HFM fine-tuning, public validation, and true 6h/12h/24h/48h/72h held-out public evaluation windows.
• Site-calibrated q-to-U protocol via scripts/run_site_adaptation_protocol.py and configs/site_adaptation.yaml: synthetic-plus-non-target-real pretraining, target-site 24h/48h calibration fine-tuning, held-out target-site heat-flux prediction, and final U computed from sum(q_hat) / sum(delta_T).
• Final 48h/72h virtual-HFM stabilization run via configs/site_adaptation_physics_block_stabilization_final.yaml: 10 seeds, four public target sites, HFM calibration on the first 48h or 72h, held-out continuation q prediction, and final U from cumulative predicted heat flux.
• 48h trace-level virtual-HFM experiment via configs/site_adaptation_physics_block_trace_48h.yaml, exporting per-timestep heat-flux predictions, sign-consistent cumulative U traces, stabilization metrics, and final findings.
• Short-window ISO/HFM baseline ISOEvalWindow, which computes U directly from measured heat flux on the same rapid evaluation window. This is the fair standard-method comparator for rapid-duration claims.
• Windowed 12h PhysicsTransformer HPO via configs/mdpi_transfer_hpo_windowed12_physics.yaml, including validation-reference objective support and separate synthetic/public direct-U regularization weights.
• Deterministic window subsampling for redundant overlapping synthetic windows, plus long-duration public split overrides so 72h cases have full calibration windows before held-out evaluation.
• Kept ablation configs for rejected/manual diagnostics, including PhysicsTransformer U-head, selected Optuna trial replay, and HFM-feature input checks.
• ISO 9869-like, linear regression, LSTM, Transformer, and Physics-Informed Transformer comparisons.
• Physics-informed heat-flux loss: data loss, transient RC residual q_hat ~= U * delta_T + C * d(delta_T)/dt + bias, smoothness, and an RC-prior residual Transformer path for public HFM.
• Series-holdout train/validation/test split, so test windows are from unknown wall instances.
• Split-conformal MC-dropout uncertainty intervals with raw and calibrated coverage, width, normalized width, miss-side rates, and Winkler score.
• Paper report artifacts in reports/.
• Docker CUDA workflow, MLflow tracking, tests, Ruff, CI, FastAPI inference, and Prometheus metrics.

Benchmark note: current synthetic results are direct U-value regression on unknown multilayer walls. They are not an apples-to-apples reproduction of Gumbarevic et al., Energies 2022, 15, 5029, which forecasts heat flux and then computes U with ISO 9869 averaging. The alignment plan and public real-data sources are documented in public_real_data_and_mdpi_alignment.md.

The repo now keeps seven tracks:

• Direct rapid U estimation: short-window unknown-wall task in reports/paper_results.md.
• MDPI-style heat-flux forecasting: q-forecast task in reports/mdpi_protocol_results.md.
• Sim-to-real heat-flux transfer: synthetic pretraining plus public-HFM fine-tuning in reports/mdpi_transfer_results.md, with calibrated q-to-U HPO findings in calibrated_q_to_u_hpo_final_findings.md.
• Windowed conference sim-to-real: duration-specific public-HFM evaluation through scripts/run_conference_sim_to_real.py, where every model and ISO baseline sees the same short held-out evaluation window.
• Site-calibrated conference protocol: a realistic deployment setup in which a general heat-flux prior is adapted on the first day or two of the target site, then evaluated on the held-out target period. This is the current main paper direction; see site_adaptation_research_brief.md.
• Real-world deployment decision: quality-gated transfer acceptance, rejected selector diagnostic in reports/mdpi_transfer_realworld_solution.md, and U-aware HPO findings in mdpi_transfer_uaware_hpo_findings.md.
• Local MLOps deployment slice: q-to-U bundle, ONNX export, FastAPI serving, and smoke pipeline in reports/local_mlops_pipeline/pipeline_summary.md.

The MLOps gap analysis before NVIDIA robotics transfer is in mlops_readiness_audit.md.

Primary Public-HFM Methodology

The intended public-data method is q-to-U, not direct U regression:

temperature/time features -> model -> q_hat(t)
U_hat = sum(q_hat) / sum(delta_T)

Direct-U prediction is now treated as a synthetic benchmark or ablation only. The current public-transfer config is configs/mdpi_transfer_q_to_u_calibrated.yaml; it sets lambda_u: 0.0 and physics_u_estimator: flux, so reported u_model is calculated only from the predicted heat-flux sequence. It also adds a train-only ISO heat-flux prior and validation residual shrinkage for safer public fine-tuning. See q_to_u_methodology_correction.md.

Run the calibrated q-to-U HPO with Hydra:

docker compose run --rm training \
  python scripts/run_mdpi_transfer_hpo_hydra.py --config-name mdpi_transfer_hpo_q_to_u_calibrated

Site-Calibrated Q-To-U Protocol

The current conference pivot is site adaptation, matching the practical measurement workflow: use broad synthetic/public data as a prior, fine-tune on a short target-site HFM calibration period, then estimate held-out heat flux and compute U from the predicted heat-flux integral.

docker compose run --rm training \
  python scripts/run_site_adaptation_protocol.py \
    --config configs/site_adaptation.yaml

Fast smoke:

docker compose run --rm training \
  python scripts/run_site_adaptation_protocol.py \
    --config configs/site_adaptation.yaml --smoke

Primary deployable comparison:

• ISOCalibrationRatio: U from the same short target calibration period.
• SiteOnly*: model trained only on the target calibration period.
• Pretrained*: synthetic plus non-target-real pretraining, then target-site fine-tuning.
• ISOEvalWindow: held-out HFM averaging diagnostic, not a no-HFM deployable competitor because it uses measured heat flux on the test window.

Artifacts are written to reports/site_adaptation_probe/ by default. The working abstract and interpretation rules are in site_adaptation_research_brief.md.

Current best fixed site-adaptation run: reports/site_adaptation_lstm_w12_final uses 24h target-site calibration, a 12h model window, synthetic-plus-non-target public pretraining, and an ISO-calibrated residual heat-flux prior feature. Across 10 seeds and 5 public target sites, PretrainedLSTM has 26.469% mean reference-U error and 3.051 q-RMSE versus 34.344% and 4.016 for ISOCalibrationRatio.

This is a positive but narrow result: the LSTM improves mean reference-U error and pointwise q-RMSE, but it wins only 14/50 paired reference-U rows and 5/50 HFM-U rows. It wins 35/50 paired q-RMSE rows but only 5/50 q-integral rows, so the next research problem is cumulative flux-bias control for U = sum(q_hat) / sum(delta_T). The current claim is site-dependent residual improvement, not a broad ISO replacement. PretrainedPhysicsTransformer is not the current headline model and should remain an ablation unless later validation policy and HPO produce a stable fixed-config win.

Site-adaptation HPO and ablation findings are documented in site_adaptation_lstm_w12_final_findings.md.

Physics-informed recurrent pivot: PretrainedPhysicsLSTM is now implemented beside PretrainedPhysicsTransformer, with an additional lambda_integral_u loss targeting sum(q_hat) / sum(delta_T) inside training windows. The current physics comparison config is configs/site_adaptation_physics_lstm_transformer_integral.yaml, and the HPO config is configs/site_adaptation_physics_integral_hpo.yaml. Current probes show that PhysicsLSTM can improve pointwise q shape, but the validation-selected physics models do not yet beat ISO on deployable U estimation; see site_adaptation_physics_lstm_transformer_integral_findings.md.

Updated conservative physics result: configs/site_adaptation_physics_block_conservative_final.yaml runs ISOCalibrationRatio, PretrainedPhysicsLSTM, and PretrainedPhysicsTransformer over 10 seeds, 24h/48h target-site calibration, block-integral validation, and conservative residual shrinkage. The final 48h aggregate is positive: PretrainedPhysicsLSTM reaches 33.422% mean reference-U error and 3.662 q-RMSE, and PretrainedPhysicsTransformer reaches 35.399% and 3.675, versus 39.179% and 3.728 for ISOCalibrationRatio. The 24h result remains marginal rather than a headline: the Transformer is only 0.104 pp better than ISO on mean reference-U error, while LSTM is slightly worse on reference-U but better on q-RMSE. Full 10-seed tables, paired ISO deltas, predicted-vs-reference U values, and plots are in site_adaptation_physics_block_conservative_final_findings.md and site_adaptation_results.md.

Final 48h/72h stabilization result: configs/site_adaptation_physics_block_stabilization_final.yaml tests the practical virtual-HFM continuation use case: calibrate on the first 48h or 72h of target-site HFM, fine-tune the pretrained physics models, then predict held-out heat flux and compute U = sum(q_hat) / sum(delta_T). Across 10 seeds and four public target sites, 48h models improve pointwise q-RMSE and mean reference-U error versus ISOCalibrationRatio, but ISO remains better on the deployable HFM/integral-U metric. At 72h, ISO is best on U while physics models still reduce q-RMSE. The result is therefore not a broad ISO replacement yet; it is evidence that q-shape learning works, while cumulative flux-bias control and trace-level stabilization analysis remain the blocker. Full findings are in site_adaptation_physics_block_stabilization_final_findings.md and the generated report is in site_adaptation_results.md.

Windowed Conference Sim-To-Real Protocol

For conference rapid-duration claims, use the windowed runner. It fixes a methodological issue in earlier transfer diagnostics: changing window_hours alone changed the training windows, but final public evaluation still used the whole held-out tail. The new runner sets evaluation.eval_window_hours and evaluation.validation_window_hours, so 6h, 12h, 24h, 48h, and 72h mean real public evaluation duration, not only model input length.

docker compose run --rm training \
  python scripts/run_conference_sim_to_real.py \
    --config configs/conference_sim_to_real_q_to_u.yaml

Fast protocol smoke:

docker compose run --rm training \
  python scripts/run_conference_sim_to_real.py --smoke --no-resume

PhysicsTransformer-only 12h HPO diagnostic:

docker compose run --rm training \
  python scripts/run_mdpi_transfer_hpo.py \
    --config configs/mdpi_transfer_hpo_windowed12_physics.yaml \
    --models PhysicsTransformer --skip-final

Final v0.17 windowed conference run: 10 seeds (42-51), synthetic pretraining, public fine-tuning, public validation selection, and deployable held-out public evaluation rows. Primary metric is reference-U error because ISOEvalWindow uses measured held-out heat flux and therefore has zero HFM-relative error by construction.

Duration	Best model/baseline	Ref. U error	CI95	HFM error	q RMSE	Rows/seed
6h	ISOTrainRatio	90.462%	0.000	59.839%	3.771	5
12h	PhysicsTransformer	59.051%	15.455	123.357%	3.907	5
24h	ISOEvalWindow	25.593%	0.000	0.000%	0.000	4
48h	ISOEvalWindow	24.892%	0.000	0.000%	0.000	3
72h	ISOEvalWindow	11.599%	0.000	0.000%	0.000	3

The defensible conference claim is narrow: PhysicsTransformer is the best 12h rapid sim-to-real estimator versus reference U, improving over ISOEvalWindow by 44.258%, over ISOTrainRatio by 34.722%, and over LSTM by 45.349% in relative error reduction. It does not win 6h, where the public signal is too short, and it does not win 24h/48h/72h, where measured short-window HFM averaging has enough duration to recover. Full artifacts are in conference_sim_to_real_results.md.

Current Public-HFM Finding

The current validation-sound q-to-U public-HFM protocol is the fair comparison to the Gumbarevic-style heat-flux forecasting setup. It trains on an initial HFM period, forecasts heat flux from temperature/time features, then computes U with ISO-style averaging on q_hat. HPO compares trials only within a model family; cross-family comparison is made only in the final fixed 10-seed run.

Final calibrated residual q-to-U public-HFM benchmark, seeds 42-51. The public split is temporal within each measured wall series, so this is rapid continuation after a calibration segment, not leave-one-public-wall-out. This table evaluates the available held-out public tail after the calibration and validation split; for duration-specific rapid claims use the windowed conference runner above.

Model	Mean U error vs HFM	CI95	Mean U error vs reference	Mean q RMSE
LSTM	35.7298	7.5360	27.6393	3.1192
ISOTrainRatio	39.5059	0.0000	90.4617	3.9868
Transformer	51.1299	19.4601	73.9154	4.3307
PhysicsTransformer	66.8491	17.4680	99.0301	4.3295

Interpretation: the useful public-HFM model from this run is the calibrated LSTM residual q-to-U model. It beats ISOTrainRatio in 7 of 10 seeds, with mean HFM-relative U error reduced by 3.776 percentage points, but the paired CI95 is 7.536 points, so the improvement is modest and uncertain. It is much stronger against reference U-values. The current Transformer and PhysicsTransformer HPO results are not deployable because validation overselects residual corrections that do not generalize to held-out public evaluation segments.

The practical engineering candidate is a conservative ISO/LSTM gate: keep ISO unless val_q_RMSE(LSTM) < 0.6 * val_q_RMSE(ISOTrainRatio). On the final rows, this gives 21.520% mean HFM-relative U error and 15.823% mean reference-relative U error, selecting LSTM on 24 rows and ISO on 26 rows. Treat that as a deployment heuristic until it is selected by nested validation or a larger real dataset.

Full details are in calibrated_q_to_u_hpo_final_findings.md.

48h Trace Stabilization Experiment

The conference-focused 48h virtual-HFM trace run is:

docker compose run --rm training \
  python scripts/run_site_adaptation_protocol.py \
    --config configs/site_adaptation_physics_block_trace_48h.yaml

It keeps the fixed physics LSTM/Transformer setup but exports per-timestep q_true, q_pred, q_iso, cumulative U_hfm(t), U_model(t), and U_iso(t). The goal is to test whether, after 48h HFM calibration, the model's cumulative U estimate stabilizes enough to continue as a virtual HFM. The research plan is documented in site_adaptation_48h_trace_research_plan.md.

Final 10-seed result: this is a useful methodology result, not yet a broad physics-model win. Across four public target sites, PretrainedPhysicsLSTM improves q-RMSE in 26/40 paired seed/site cases and PretrainedPhysicsTransformer in 22/40, but ISOCalibrationRatio remains best on mean held-out HFM/integral-U error (9.908% vs 13.915% LSTM and 13.771% Transformer). Neural models slightly improve mean reference-U error (36.007% LSTM and 36.485% Transformer vs 39.179% ISO), mainly because the roof reference value conflicts strongly with the held-out HFM continuation. The paper-safe conclusion is that trace-level cumulative U analysis exposes cumulative heat-flux bias that q-RMSE alone hides. Full findings are in site_adaptation_physics_block_trace_48h_findings.md, with generated plots and CSVs in site_adaptation_physics_block_trace_48h.

Final U-value graph comparison is in final_u_value_comparison.md. For normal-gradient public rows, seed-mean U reduces the conservative ISO/LSTM gate error from 21.520% per-seed HFM-relative error to 19.063%, and reduces reference-relative error from 15.823% to 6.545%. This shows that averaging smooths the jumpy U graph, but it is a post-hoc seed ensemble diagnostic, not a single deployed model result. The dedicated predicted-vs-real reference plot is final_u_predicted_vs_real_reference_normal_gradient.png.

Heat-flux prediction diagnostics are in q_prediction_diagnostics.md. The tracked q plots are a bounded seed-42 rerun with final HPO hyperparameters, intended for shape/error inspection of q_pred vs q_true on synthetic holdout and public held-out data.

The U-aware/direct-U branch below is a rejected ablation, not the headline methodology.

Public-HFM run	Model/policy	HFM error mean	Reference error mean
HPO-selected U-aware trial 10	ISOTrainRatio	39.505907	90.461650
HPO-selected U-aware trial 10	PhysicsTransformer	84.250502	138.595819
Trial-6 stability audit	PhysicsTransformer	92.269161	148.075742
Trial-6 guarded policy, deploy Physics only when validation U error <= 1%	ISO fallback + PhysicsTransformer	37.438397	82.901895

Interpretation: do not claim a pure PhysicsTransformer public-HFM win yet. The credible current result is a guarded deployment policy: use PhysicsTransformer only when validation confidence is very strong, otherwise fall back to ISO. Full details are in mdpi_transfer_uaware_hpo_findings.md.

Paper Results

Synthetic corpus: data/synthetic/paper_v2, 480 wall series, dataset hash:

49cff46c3746e51eac991926beae110d8d6c43b3a750a7c4337e78a047b81d94

Three-seed duration grid: seeds 42, 43, 44; durations 6h, 12h, 24h, 48h, 72h.

Corpus coverage:

• 6 wall archetypes: cavity wall, concrete sandwich, insulated masonry, lightweight frame, solid masonry, timber frame.
• 5 outdoor profiles: cold front, shoulder, solar swing, warm spell, winter.
• Explicit layer stacks: 3 to 5 material layers per wall.
• Heat-flux noise bins by series: low 79, medium 136, high 265.
• Solar disturbance bins by series: low 102, medium 159, high 219.

Best model by duration:

Duration	Best model	MAE mean	RMSE mean	R2 mean
6h	PhysicsTransformer	0.185457	0.294486	0.751561
12h	PhysicsTransformer	0.156856	0.248601	0.823902
24h	PhysicsTransformer	0.121325	0.188511	0.897576
48h	LSTM	0.105865	0.158291	0.927966
72h	ISO	0.105546	0.189260	0.862404

Interpretation: the Physics-Informed Transformer is strongest for rapid 6h, 12h, and 24h unknown-wall windows, which is the abstract's target regime. For longer 48h/72h windows, averaging has enough data to suppress transient noise, so ISO and LSTM are very competitive. The repository does not claim transformer superiority where long-duration HFM averaging is already effective.

High heat-flux-noise slice:

Duration	Best model	MAE mean	RMSE mean	R2 mean
6h	Transformer	0.197028	0.298821	0.771667
12h	PhysicsTransformer	0.167825	0.254436	0.834255
24h	ISO	0.122538	0.197315	0.898085
48h	ISO	0.109155	0.162157	0.930171
72h	ISO	0.100681	0.143271	0.944767

Short noisy-window behavior: ISO is weak at 6h/12h under high heat-flux noise (6h MAE=0.353807, 12h MAE=0.292539). Transformer-family models reduce that error substantially (6h best 0.197028, 12h best 0.167825). At 24h and beyond, ISO averaging recovers because the measurement window is long enough.

Fresh rapid uncertainty check: PhysicsTransformer_12h_seed44.pt with validation-calibrated MC dropout produced PI_Coverage=0.921104, PI_MeanWidth=0.948390.

Generated artifacts:

• paper_results.md
• metrics.csv
• metrics_by_duration.csv
• slice_metrics.csv
• slice_metrics_by_condition.csv
• error_vs_duration.png
• leaderboard.png
• uncertainty_intervals.png

Setup

Use Docker for the main workflow because host Python/CUDA compatibility can drift.

docker compose build training

Native setup, if your Python version is supported:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

Native Apple Silicon / MPS

On an arm64 Mac, use a project-local uv environment (the tested M5 setup uses Python 3.13):

uname -m  # arm64
uv sync --extra dev --python 3.13

The M5-generated uv.lock is only for this native Apple Silicon workflow; it is not a Linux/CUDA migration path. The existing remote CUDA workflow remains Docker-based on pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime. Do not reuse the M5 environment to replace or update that image.

Do not enable PYTORCH_ENABLE_MPS_FALLBACK: an unsupported MPS operation should fail visibly rather than silently run on CPU. Use conservative unified-memory watermarks, with the low ratio no greater than the high ratio and never set the high ratio to zero:

unset PYTORCH_ENABLE_MPS_FALLBACK
export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.85
export PYTORCH_MPS_LOW_WATERMARK_RATIO=0.75

Run the bounded, identical CPU and MPS probes in separate processes:

uv run python scripts/benchmark_site_adaptation_m5.py --device cpu
uv run python scripts/benchmark_site_adaptation_m5.py --device mps

For a less startup-sensitive bounded comparison, run three independent processes per device. Safe --run-id slugs create separate output directories instead of overwriting earlier probes:

for run_id in r1 r2 r3; do
  env -u PYTORCH_ENABLE_MPS_FALLBACK \
    uv run python scripts/benchmark_site_adaptation_m5.py \
      --device cpu --run-id "$run_id"
done

for run_id in r1 r2 r3; do
  env -u PYTORCH_ENABLE_MPS_FALLBACK \
    PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.85 \
    PYTORCH_MPS_LOW_WATERMARK_RATIO=0.75 \
    uv run python scripts/benchmark_site_adaptation_m5.py \
      --device mps --run-id "$run_id"
done

uv run python scripts/benchmark_site_adaptation_m5.py \
  --summarize-run-ids r1 r2 r3

Probe outputs are isolated under reports/m5_probe_cpu/ and reports/m5_probe_mps/. The benchmark records wall time, process peak RSS, and, for MPS, sampled current/driver allocations and the recommended maximum. macOS Activity Monitor reports whole-system unified-memory pressure and may not match PyTorch allocator counters or process RSS exactly.

The full MPS config preserves the causal 48h config's seeds, models, training batch sizes, and scientific hyperparameters. A full 10-seed run is long and is not recommended as a thermal probe on a fanless MacBook Air:

mkdir -p mlartifacts_m5
uv run mlflow server \
  --backend-store-uri "sqlite:///$PWD/mlflow_m5.db" \
  --artifacts-destination "$PWD/mlartifacts_m5" \
  --host 127.0.0.1 --port 5001 --workers 1 \
  --allowed-hosts "localhost:*,127.0.0.1:*"

Open http://127.0.0.1:5001, then launch the tracked float32 run in another terminal. The config logs CPU and unified-memory system metrics every 10 s; the benchmark wrapper also records sampled MPS allocator peaks:

env -u PYTORCH_ENABLE_MPS_FALLBACK \
  PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.85 \
  PYTORCH_MPS_LOW_WATERMARK_RATIO=0.75 \
  uv run python scripts/benchmark_site_adaptation_m5.py \
    --device mps --full --mlflow \
    --config configs/site_adaptation_physics_block_trace_48h_causal_mps.yaml

If the probes select CPU, this command runs the unchanged full causal workload without overwriting the CUDA, MPS, or paper reports (output uses the _m5_cpu profile):

uv run python scripts/benchmark_site_adaptation_m5.py --device cpu --full

MPS, CUDA, and CPU results are not expected to be bitwise identical. This path uses float32 only; mixed precision is not enabled without a separate performance and numerical validation. Backend differences can also change validation block scores and the selected residual alpha, which can alter downstream metrics and scientific conclusions. Compare the per-row parity fields; strict backend equality is not a probe requirement. Any timing statement applies only to this bounded workload and machine, not to MPS performance in general.

Generate Synthetic Data

Paper corpus:

python scripts/generate_synthetic.py --config configs/synthetic.yaml

Smoke corpus:

python scripts/generate_synthetic.py --config configs/synthetic_smoke.yaml

Train

python scripts/train.py --config configs/physics_transformer.yaml

GPU on this Fedora host uses plain Docker:

docker run --rm --gpus all \
  -v "$PWD":/app -w /app \
  uvalue-training:local \
  python scripts/train.py --config configs/physics_transformer.yaml

Smoke CPU:

python scripts/train.py --config configs/smoke_cpu.yaml

Evaluate

python scripts/evaluate.py \
  --checkpoint artifacts/models/best_model.pt \
  --config configs/evaluate.yaml

Evaluation includes calibrated MC-dropout intervals when uncertainty.enabled is true.

Run Paper Grid

One-seed quick grid:

python scripts/run_experiment_grid.py --config configs/experiment_grid.yaml

Three-seed paper grid:

docker run --rm --gpus all \
  -v "$PWD":/app -w /app \
  uvalue-training:local \
  python scripts/run_experiment_grid.py --config configs/paper_grid.yaml

Run MDPI-Style Protocol

This is separate from the direct-U benchmark. It trains on an initial HFM period, forecasts heat flux from temperature features, and computes U by ISO-style averaging.

python scripts/run_mdpi_protocol.py --config configs/mdpi_protocol.yaml

Historical standalone MDPI protocol artifacts are kept below for auditability. The accepted current public-HFM result is the calibrated residual q-to-U HPO run summarized in calibrated_q_to_u_hpo_final_findings.md.

Report artifacts:

• mdpi_protocol_results.md
• mdpi_protocol_metrics.csv
• mdpi_protocol_10seed_results.md
• mdpi_protocol_10seed_metrics.csv

Run Synthetic-to-Public Transfer Protocol

This is the stricter sim-to-real workflow: pretrain neural heat-flux forecasters on synthetic multilayer RC-wall series, select on synthetic validation, fine-tune on public HFM train segments, select on public validation segments, and report public test segments only. ISOTrainRatio uses the public train segment as a non-neural baseline.

docker compose run --rm training \
  python scripts/run_mdpi_transfer.py --config configs/mdpi_transfer.yaml

Historical long single-seed GPU transfer diagnostic: seed 42, 12h windows, synthetic corpus data/synthetic/paper_v2, public corpus data/raw/hfm_public, 200 synthetic pretraining epochs, 300 public fine-tuning epochs, and 40 train+validation refit epochs.

Model	Mean U relative error vs HFM	Median U relative error vs HFM	Mean q RMSE
ISOTrainRatio	78.834%	20.516%	8.047
LSTM	79.654%	29.583%	4.138
Transformer	202.449%	163.592%	6.269
PhysicsTransformer	710.001%	417.959%	6.452

Normal-gradient subset (mean |delta_T| >= 3 K):

Model	Mean U relative error vs HFM
LSTM	27.165%
ISOTrainRatio	39.506%
Transformer	89.112%
PhysicsTransformer	239.618%

Interpretation: longer training greatly improved LSTM heat-flux forecasting and normal-gradient U error. Transformer-family heat-flux RMSE also improved, but U aggregation remains unstable, especially on low-gradient public cases. The PhysicsTransformer result is contaminated by the long train+validation refit step, which overwrote the best validation checkpoint and hurt U estimation. For cleaner curve analysis, configs/mdpi_transfer_transformers_xlong.yaml runs only Transformer and PhysicsTransformer for longer budgets and disables the final refit, so final evaluation uses the best public-validation checkpoint.

Completed extra-long Transformer-family diagnostic: seed 42, 12h windows, 336 synthetic train series, 72 synthetic validation series, 72 synthetic holdout series, 7 public HFM series, 600 synthetic pretraining epochs, 900 public fine-tuning epochs, CUDA device, and no final refit.

Model	Mean U relative error vs HFM	Median U relative error vs HFM	Mean q RMSE
Transformer	142.092%	83.013%	4.516
PhysicsTransformer	208.790%	77.853%	8.370

Normal-gradient subset (mean |delta_T| >= 3 K):

Model	Mean U relative error vs HFM	Mean q RMSE
PhysicsTransformer	55.654%	6.010
Transformer	78.433%	4.137

Low-gradient subset (mean |delta_T| < 3 K):

Model	Mean U relative error vs HFM	Mean q RMSE
Transformer	301.239%	5.463
PhysicsTransformer	591.630%	14.268

Reference-U caveat for low-gradient public cases: the public HFM-derived test U can diverge strongly from the provided reference U under weak temperature gradients. On the two low-gradient cases, the plain Transformer has 301.239% mean error versus the HFM-derived test U but 8.855% mean error versus the provided reference U. Those rows should be discussed as an HFM target-stability problem, not only as model failure.

Curve interpretation: the plain Transformer benefited from longer public fine-tuning; its public validation score was best at epoch 848/900, and its public q RMSE improved from 6.269 in the long run to 4.516. The PhysicsTransformer no-refit run fixed the previous catastrophic refit result (710.001% mean U error down to 208.790%), but its public validation score was already best at epoch 15/900 and the physics loss later increased. That points to overfitting or a mismatched physics penalty on public HFM, not simply an insufficient training budget. Low-gradient public cases remain the main failure mode because U estimation becomes ill-conditioned when the evaluation temperature difference is weak.

Report artifacts:

• mdpi_transfer_results.md
• mdpi_transfer_metrics.csv
• mdpi_transfer_config.yaml
• mdpi_transfer_public_error.png
• mdpi_transfer_public_error_by_series.png
• mdpi_transfer_q_rmse.png
• mdpi_transfer_transformers_xlong/results.md
• mdpi_transfer_transformers_xlong/metrics.csv
• mdpi_transfer_transformers_xlong/public_error.png
• mdpi_transfer_transformers_xlong/public_error_by_series.png
• mdpi_transfer_transformers_xlong/q_rmse.png

Real-World Transfer Decision

The accepted real-world protocol is quality-gated transfer, not blind model selection. The current calibrated q-to-U protocol keeps ISOTrainRatio, LSTM, Transformer, and PhysicsTransformer in the same tracked experiment, but a U-value is deployable only when mean |delta_T| >= 3 K; low-gradient periods are marked defer_low_gradient.

Current accepted deployed public-test result, calibrated q-to-U protocol, normal-gradient cases only (5/5 prepared public series):

Model	Mean U error vs HFM	Median U error vs HFM	Mean U error vs reference	Mean q RMSE
Conservative ISO/LSTM gate	21.520%	19.374%	15.823%	3.248
LSTM	35.730%	28.718%	27.639%	3.119
ISOTrainRatio	39.506%	19.374%	90.462%	3.987
Transformer	51.130%	46.273%	73.915%	4.331
PhysicsTransformer	66.849%	72.180%	99.030%	4.330

Decision: the conservative ISO/LSTM gate is the current public-HFM engineering candidate. The ungated calibrated LSTM remains the best single neural model, but ISOTrainRatio stays as the fallback because the LSTM HFM-relative improvement is modest (-3.776 paired percentage points vs ISO, CI95 7.536). Transformer and PhysicsTransformer should run as shadow estimators only until a more robust unknown-wall public benchmark is added.

Validation model selection across all four model families was tested and rejected for the calibrated run. It worsened mean HFM error to 57.509% because it overselected unstable Transformer and PhysicsTransformer residuals.

A conservative ISO/LSTM gate is the current deployable candidate: choose LSTM only when validation q RMSE is less than 0.6 * ISOTrainRatio validation q RMSE, otherwise keep ISO. On the completed final rows this diagnostic gives 21.520% mean held-out HFM U error with 4.455% CI95, selecting LSTM for 24 rows and ISO for 26 rows. The threshold should be validated by nested selection or more real campaigns before it becomes a paper claim.

Report:

• mdpi_transfer_realworld_solution.md
• calibrated_q_to_u_hpo_final_findings.md

Real HFM Retraining

Place labeled real HFM CSV files under data/raw/hfm/.

Validate:

python scripts/validate_real_data.py --path data/raw/hfm

Fine-tune from the synthetic checkpoint:

python scripts/retrain_real.py --config configs/real_retrain.yaml

Expected canonical columns:

series_id,timestamp,T_in,T_out,delta_T,q,solar,u_value

If series_id is missing, each CSV is treated as one series. If delta_T is missing, it is computed. If solar is missing, it is set to 0.0.

Public DOI HFM Data

Download DOI-indexed public files from Mendeley Data:

python scripts/download_public_hfm.py --output-root data/raw/public

Prepare canonical real HFM CSVs:

python scripts/prepare_public_hfm.py \
  --raw-root data/raw/public \
  --output-root data/raw/hfm_public

Validate and fine-tune:

python scripts/validate_real_data.py --path data/raw/hfm_public
python scripts/retrain_real.py --config configs/public_real_retrain.yaml

Known public sources:

Dataset key	DOI	Notes
mendeley_block_f_v3	10.17632/xvhpnpnr3w.3	Wall HFM CSV with heat flux, temperatures, and U column.
mendeley_degradation_v1	10.17632/4kbb93bx32.1	Multiple HFM CSVs for envelope components.
mendeley_greenway_v1	10.17632/3gms2zvykb.1	Workbook with heat flux and temperature series; U label is ISO-derived by the adapter.

MLflow

Start tracking server:

docker compose up -d mlflow

UI:

http://localhost:5001

Logged fields include model type, window duration, seed, dataset hash, Git commit hash, losses, MAE, RMSE, MAPE, R2, calibrated interval coverage, interval width, config artifact, prediction plots, and checkpoints. The synthetic-to-public transfer protocol also logs synthetic/public dataset hashes, split counts, device, pretraining/fine-tuning/refit epoch losses, validation selection scores, and final public-test aggregate metrics under experiment physics_informed_u_value_mdpi_transfer.

Inference Service

Build the deployable calibrated q-to-U bundle from selected HPO parameters:

docker compose run --rm training \
  python scripts/train_q_to_u_bundle.py \
    --output-bundle artifacts/q_to_u/calibrated_lstm_q_to_u.pt

Evaluate and export the trusted path:

docker compose run --rm training \
  python scripts/evaluate_q_to_u_bundle.py \
    --bundle artifacts/q_to_u/calibrated_lstm_q_to_u.pt

docker compose run --rm training \
  python scripts/export_q_to_u_onnx.py \
    --bundle artifacts/q_to_u/calibrated_lstm_q_to_u.pt \
    --output artifacts/q_to_u/calibrated_lstm_q_forecaster.onnx

Run the local train/evaluate/export/serve smoke pipeline:

docker compose run --rm training python scripts/run_local_mlops_pipeline.py

The smoke run writes a compact report to reports/local_mlops_pipeline/pipeline_summary.md. It is an operational pipeline check, not a replacement for the full 10-seed paper benchmark.

docker compose up inference

For the existing homelab Prometheus/Grafana stack on coder, publish the service on the laptop Tailnet IP:

UVALUE_INFERENCE_BIND_IP=100.64.0.2 docker compose up inference

If artifacts/models/best_model.pt exists, the service loads model dimensions, feature order, and normalization metadata from the checkpoint. If artifacts/q_to_u/calibrated_lstm_q_to_u.pt exists, the same service also loads the calibrated q-to-U bundle.

Endpoints:

GET  /health
GET  /model-info
POST /predict
POST /estimate-u
GET  /metrics

POST /predict serves the older direct-U checkpoint path. POST /estimate-u is the corrected public-HFM path: it accepts calibration measurements, optional validation measurements, and evaluation measurements; predicts heat flux; computes U with ISO-style averaging; and returns the selected deployment decision.

Observability

Local metrics commands:

nvidia-smi
nvidia-smi dmon
nvidia-smi pmon
docker stats
curl -s localhost:8000/metrics

Compose services include Prometheus and Grafana placeholders:

docker compose up prometheus grafana

Quality Gates

ruff check .
ruff format --check .
pytest

Docker:

docker compose run --rm training pytest
docker compose run --rm training ruff check .
docker compose run --rm training ruff format --check .

Robotics Handoff

The same discipline maps to NVIDIA robotics onboarding:

• RC-wall simulator -> Isaac Sim / Isaac Lab simulation.
• Synthetic pretraining -> sim-to-real policy pretraining.
• Sensor noise -> robot sensor noise and domain randomization.
• Physics loss -> safety constraints and reward shaping.
• MLflow grid -> policy experiment tracking.
• FastAPI inference -> ROS 2 node or Triton service.
• Prometheus/Grafana -> robot/fleet observability.
• Docker GPU workflows -> Jetson deployment workflows.

See ROBOTICS_HANDOFF.md.

References

• Logibot: https://www.logibot.eu/
• ISO 9869-1 heat flow meter method: https://www.iso.org/standard/59697.html
• Attention Is All You Need: https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf
• Physics-informed neural networks: https://www.sciencedirect.com/science/article/pii/S0021999118307125
• PyTorch: https://docs.pytorch.org/tutorials/
• MLflow Tracking: https://mlflow.org/docs/latest/ml/tracking
• Docker GPU Compose: https://docs.docker.com/compose/gpu-support/
• ROS 2 concepts: https://docs.ros.org/en/rolling/Concepts/Basic.html
• Isaac Sim: https://docs.isaacsim.omniverse.nvidia.com/
• Isaac ROS: https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-ros/latest/
• TensorRT: https://docs.nvidia.com/deeplearning/tensorrt/latest/
• Triton: https://docs.nvidia.com/deeplearning/triton-inference-server/