Skip to content

Include information about model performance in easier to access format #3

@BryanBaird

Description

@BryanBaird

I think this is something that can be calculated from existing data components, but might not currently be packaged up in one consistent format.

Here were some of the analysis questions I encountered that I could not immediately figure out how to address with the current data structure:

  • Was the effect of X holiday larger or smaller this year than prior year(s)? By how much? (In absolute count and in relative percent)
  • When comparing trends (i.e. across multiple dates), is the gap between forecast and observed growing, shrinking, or staying level?
    • E.g., if a model underestimated holiday impact, do the trends re-converge in the following days or does that gap persist as a step-change?
  • Within each country/population, how far off was the forecast value from the observed value on a given date? (In absolute count and relative percent)
  • What exactly was the impact of holiday(s) on the forecast value for a given date? (In absolute count and relative percent)
    • This may just be the difference between the detrended forecast and the final forecast.
  • What was the estimated impact of holiday(s) on a historical observed value for a given date? (In absolute count and relative percent)
    • This may just be the difference between the detrended amount and the observed amount.

Ideally, this could be included in a simple DataFrame -- either as additional columns in the same general DataFrame output from the general to_df() function, or maybe as a separate dedicated function for performance_df().

(Also acknowledging that this might be different than some other object that stores overall measures of model performance -- things that might come out of Prophet's performance_metrics).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions