4.1.3 Data Loading

The data we get and transform needs to be stored.

There are several questions that need to be addressed in order to do it:

  • At which levels we store which data?
    • Do I need to transfer all the data to the cloud level, or it’s enough if I store some of it, as for example the raw captures, at edge or fog level?
  • How do we define the database structure?
    • Closed, as a classical relational database?
    • Open, as in NoSQL environments, able to accept non structured data sources?
  • What data should we store?
    • Do we need to keep the raw data for verification purposes?
    • Should we also keep images of the system status every X minutes, or cycles?
    • Do I record the accumulated information? By second, hour, day, month?
  • What database format is best for my purposesSQL?
    • NoSQL?
    • Parquet?
    • Timeseries optimized formats?
  • How do I incorporate new data?
  • For how long should we keep any of those records?