For demand forecasting, training must see only information available before the forecast is released. That is why data are split by time rather than by randomly shuffling rows. It is also important to check when features became available and when the target outcome became fully known. A gap between training and validation helps only when it matches those timing constraints.
This checklist is intended for an analyst and the owner of the inventory replenishment process. The result is a scheme for training, validation, and test periods that can be applied to your own dataset. The official documentation was reviewed on November 4, 2026. The date example below is fictional; no model training, tuning, or accuracy measurement was performed in this article.
Contents
- 1. Define the decision point and forecast horizon
- 2. Separate the event date from the availability date
- 3. Remove incomplete targets from training
- 4. Hold out the final period for tuning
- 5. Build sequential validation windows
- 6. Check what gap means in your table
- 7. Fit transformations within each window
- 8. Save boundaries and constraints with the results
1. Define the decision point and forecast horizon
- Record the time when the company issues the forecast.
- Specify which future dates or intervals must be predicted.
- Define whether you are forecasting daily sales, a total over a horizon, or another measure.
Suggested example: on the morning of October 1, a planner forecasts total sales from October 1 through October 7 inclusive. In that case, the horizon covers seven calendar days. The history used for the decision must reflect the data available that morning, not the state of a data extract received at the end of the month.
Stage result: one description of the forecast moment, target interval, and unit of measurement. If the sales team understands the horizon as a week and the analyst as the next row in the table, resolve that difference first.
2. Separate the event date from the availability date
- For each source, clarify when the event occurred.
- Also determine when the record became available to the forecasting system.
- For correctable data, determine whether the historical version can be reconstructed.
A payment may belong to Monday but reach the data mart later. The final amount for Monday cannot be used automatically when modeling a decision made before it arrived. This is a practical application of the principle that validation data must not tell training something that was not yet known at the time of the real decision.
The scikit-learn guide on data leakage separately discusses the improper use of information that is unavailable at prediction time. Here we apply that principle to the timing of business data arrival; the table below is not a built-in library function.
| Feature | Availability question | Suggested action |
|---|---|---|
| Yesterday's sales | Are they closed by morning? | Use only the available version |
| Planned promotion | Was the promotion approved in advance? | Store the plan version as of the decision date |
| Actual discount tomorrow | Is it known now? | Do not substitute a future observation |
| Ending balance for next week | Will the value exist after the forecast? | Exclude as an unavailable feature |
Stage result: for each feature, there is a rule for whether it is allowed at the decision point. If historical availability cannot be reconstructed, that limitation should be preserved in the conclusions rather than hidden in the overall metric.
3. Remove incomplete targets from training
- Clarify when the target interval for each training record ends.
- Add the delay until the final data arrival, if there is one.
- Exclude records whose target outcome was not yet known on the training date.
In the training example, the model is released on the morning of October 1. If the target for a record is the sum over seven days from the start date, then the record starting on September 30 requires observations through October 6. On October 1, it is not yet suitable as a completed training target. The last potentially valid start date without an arrival delay is September 24: its interval ends on September 30. If the data are closed later, the boundary shifts earlier.
This is a calculation based on the specified dates, not a universal rule to “leave a week out.” If you are training a model for daily sales with one row per target day, the target structure is different. First define what the row means, then choose the gap.
Stage result: a target maturity rule. For each allowed training record, the end of the target interval and the time when the result became available are known.
4. Hold out the final period for tuning
- Set aside a final test period that mimics future production use.
- Use earlier periods for development and validation.
- Do not tune features or hyperparameters based on the final test error.
The purpose of the final test is to evaluate an approach that has already been selected. If you change the model after seeing the result, that period is effectively part of development. A new independent evaluation would require another holdout period, or an honest statement that no independent test remains.
For time series, the documentation for TimeSeriesSplit describes sequential splits: training occurs before the validation block. That does not automatically forbid using the final test set; the team defines the organizational boundary between development and final evaluation.
Stage result: fixed dates for the three parts and a clear purpose for each. Choose the sizes based on forecast frequency, seasonality, and the amount of history available; the same share of rows does not fit every task.
5. Build sequential validation windows
- Choose several forecast release moments within the development period.
- For each one, reconstruct the available history and the completed targets.
- Compare the model with a baseline approach on the same dates and horizons.
A suggested setup is first a forecast at the beginning of September using earlier history, then another at the beginning of October with extended history. If the system is retrained once a month, daily retraining during validation will model a different process. Specify whether model parameters are updated between windows or only the input features change.
The official example of forecasting with lag features compares random and time-based splitting. The example shows why the evaluation method matters; its results cannot be transferred to a company’s assortment as an expected accuracy gain.
Stage result: a list of windows with the training date and the start and end of validation. For each window, you can explain why the information used was available at that moment.
6. Check what gap means in your table
- Determine whether a row represents a day, a product for a day, or the start of a forecast window.
- Check the row order and the regularity of the time grid.
- Do not convert a number of rows into calendar days without checking.
The parameter gap in TimeSeriesSplit sets the number of observations excluded from the end of the training set before validation. It does not mean calendar duration. The documentation also notes the regular spacing of observations over time for comparable test-block duration.
If a single day contains many products, seven rows do not equal seven days. For this kind of panel, the recommended approach is to first set boundaries by unique dates, then assign all records for the same day to one segment. Then separately verify target maturity and feature availability. The name of a splitting function alone does not prove these conditions.
Stage result: The gap is reflected both in the time rules and in the actual records. If dates are missing, first decide whether they mean no observation, a closed store, or zero sales; these situations cannot be silently treated as the same.
7. Train transformations within each window
- Calculate trainable gap-filling parameters only on the training portion.
- Likewise, limit scaling, feature selection, and encoding.
- Apply the parameters already found to the validation data.
The overall average across all dates uses future information, even if the model is later trained only on older rows. A pipeline helps connect transformation training and model training, but it does not fix features that were built in advance with future values. The causes and ways to prevent this are described in the common pitfalls section.
For moving averages, record which source dates go into each value and when those observations become available. Calling a feature “lagged” is not enough: averaging that accidentally includes the day being forecast changes the problem setup.
Stage result: The feature-building and transformation process is repeated separately for each window. Check question: can you obtain this feature vector using only the historical snapshot as of the decision date?
8. Save boundaries and constraints together with the results
- Save the dates and record IDs for each split.
- Record the horizon, source latency, and target maturity rules.
- Show the error by individual windows and the relevant business segments.
This is a proposed reproducibility package, not a report on an actual experiment. Separately indicate the products for which the history is insufficient and the periods when the accounting method changed. Do not combine all windows into one number if it hides a failure in the last period.
The broader context of data, metrics, and the role of forecasting is covered in the article “Demand Forecasting with AI” . This checklist addresses a narrower task: which records are allowed when reproducing a specific decision moment.
Stage result: Another analyst can reconstruct the same splits and explain the origin of each feature. This still does not prove the usefulness of the purchasing decision: that is evaluated separately.
How Vladimir Lomtev’s AI Studio helps prepare forecasting data
Vladimir Lomtev’s AI Studio handles data preparation, ML forecasting, and testing. For time-based splitting, the work may include describing source availability, building historical snapshots, and comparing forecasts on the same time windows.
Start with one replenishment decision: specify the current planning method, sources, forecast release time, constraints, and the metric for evaluating it. Without these conditions, even a careful split may test a different problem. Inventory reduction or errors are not guaranteed in advance.
Bottom line: signs of a proper time-based split
The split is ready when training does not use future features or unfinished targets, all products from the same reference day fall into a consistent period, and the final test is separated from tuning. Check this through the decision date and information availability. A fixed row share and an arbitrary gap do not replace such a check.