4.1.3 Data Loading

The data we get and transform needs to be stored.

There are several questions that need to be addressed in order to do it:

• At which levels we store which data?
  • Do I need to transfer all the data to the cloud level, or it’s enough if I store some of it, as for example the raw captures, at edge or fog level?
• How do we define the database structure?
  • Closed, as a classical relational database?
  • Open, as in NoSQL environments, able to accept non structured data sources?
• What data should we store?
  • Do we need to keep the raw data for verification purposes?
  • Should we also keep images of the system status every X minutes, or cycles?
  • Do I record the accumulated information? By second, hour, day, month?
• What database format is best for my purposesSQL?
  • NoSQL?
  • Parquet?
  • Timeseries optimized formats?
• How do I incorporate new data?
• For how long should we keep any of those records?