I am creating a couple of bigger database tables with at least hundreds of millions of observations, but growing. Some tables are by minute, some by milliseconds. timestamps are not necessarily unique.
Should I create separate year, month, or date and time columns? Is one unique datetime column enough? At what size would you partition the tables?
Raw data is in csv.
Currently I aim for postgres and duckdb. Does timescaledb make a significant difference?
I thought InfluxDB is the choice for such use cases. But I’m not an expert…