[OPINION] InfluxDB OSS : long term plan to migrate towards a different time series database

[arpanghosh8453]

With the release of InfluxDB 3.0, several architectural changes have introduced limitations that affect self-hosted projects like this one. Notably:

• Query Time Range Limitation: InfluxDB 3.0 Core restricts individual queries to a 72-hour time range. This constraint is due to performance considerations related to the number of Parquet files involved in query planning and execution. Queries spanning longer periods may lead to increased memory usage and potential out-of-memory errors. As we are interested in long term visualization and using InfluxDB, this is very concerning.

• Flux Language Introduction And Deprecation: The Flux query language, introduced in InfluxDB 2.0, is now in maintenance mode. InfluxDB 3.0 does not support Flux, focusing instead on SQL and InfluxQL. This kind of backward-incompatible frequent switches makes projects like this harder to maintain. It also points out the instability of the decision made by the InfluxDB team over time - We can't consider this as a stable software anymore.

• Focus on Recent Data: InfluxDB 3.0 Core is optimized for recent data and does not include features like a compactor for reorganizing data for deletes or query optimization over longer time periods. This design choice makes it less suitable for workloads requiring efficient queries over historical data.

---

With these changes, it seems like InfluxDB has decided to take a corporate stance focusing only on the profit for the company. As that does not go hand in hand with the OSS model, given these changes, it's prudent to consider alternative time-series databases that better align with the needs of this project.

Potential alternatives include:

• QuestDB: A close alternative, going through rapid developments. Supports InfluxDB line write formats and data migrations from InfluxDB. Can be a drop in replacement, but the Grafana panels needs to be converted in SQL queries. This seems like the most promising option available so far in my research.

• VictoriaMetrics: A fast, cost-effective, and scalable time-series database. A very good future alternative. can be a drop in replacement as it already supports line series data write (compatible with Influxdb v1.x). this has already been mentioned and requested in [FEATURE] Support Victoriametrics/Prometheus backend. #36. It does not support multiple database structure (measurements are stored in the global namespace) and string fields, which are not a major concern for this project.

• TimescaleDB: Built on PostgreSQL, offering robust time-series capabilities and SQL support. Requires additional changes in code for read write compatibility. Relational database, but we won't be using that here.

Although this does not require any immediate action, until InfluxDB v1.x gets the unmaintained/deprecated status, it is important to be aware of this and plan ahead. I Would like to get thoughts of people who are more experienced in database structures. Feel free to comment your thoughts here as we are looking for user's opinion on this thread.

[arpanghosh8453]

Hello everyone, I just tested this with QuestDB backend, and it seems to be a drop in replacement due to line protocol compatibility. Just need to change the hostname and port in the ENV variables and it worked!

This will require a new dashboard JSON for Grafana of course because it does not support InfluxQL (only SQL is supported)

Just replacing the influxdb container with this in the stack should do it. You need to also update the depends_on to questdb replacing influxdb in the compose file and update the hostname and port in garmin-fetch-data compose config. Here I have mapped the 9000 port to 8086, but that's optional as long as you point the garmin-fetch-data container to the right QuestDB port.

services:
  questdb:
    ports:
      - 8086:9000
    image: questdb/questdb:8.3.0
    container_name: questdb
    hostname: questdb
    restart: unless-stopped
    volumes:
      - "./database:/var/lib/questdb"

This is an extension of issue #36

Another issue is I can't get the data to deduplicate properly when fetched twice. InfluxDB does this automatically using the time and tags for the primary key hash for each measurement. Here, I was not able to figure out how to achieve the same. With the above setup, it's working as append only :(

[M0dEx]

Ah, it has to be manually enabled for the tables using ALTER TABLE <table_name> DEDUP ENABLE UPSERT KEYS(timestamp).

[arpanghosh8453]

@M0dEx I did check that as well, and I wanted to use the tag keys combined with the timestamp for the deduplication for good measures, as there might be two data at the same timestamp, for different devices on the same table.

[M0dEx]

That should be possible as well, but the downside is that it needs to be manually set-up for every table.

[arpanghosh8453]

@M0dEx I believe we can do those with automated script commands during database setup. We know the table names, so should be possible.

[M0dEx]

That's true. Given that, the worst part of this process will probably be updating the dashboard, as it needs to be done manually.

[depau]

Hi! I'm not an expert but I think Mimir might also be an option: https://grafana.com/oss/mimir/
It has an Influx-compatible ingestion API

[meron1122]

You can abstract storage layer. This allow users to choose own solution. Dashboards is problematic, but I think not mission impossible task.

[DaanHend]

I’d also like to give a +1 for TimescaleDB as an alternative. More a personal favorite though.

Being built on top of PostgreSQL it makes it a very stable and well-supported option in the long term. It provides good time-series capabilities (continuous aggregates, compression, retention policies)

It probably requires more adaptation effort up front (because of Influx line protocol? Never used influx and setting up the schemes beforehand I would imagine), but you got SQL, and the database is based on a widely adopted, battle-tested technology.

I havent looked into the code yet though. So just an comment to keep the discussion burning

[arpanghosh8453]

@DaanHend The data is currently processed as influx line protocol but we can write a quick wrapper function which can parse the data and make it compatible for PostgresSQL writing - and I agree on all the points you stated here - The main challenge comes redesigning the dashboard - because all the queries needs to be ported and that will be a major work tbh. I will be happy to adopt TimescaleDB otherwise in a heartbeat.

[depau]

I thought I already mentioned it but I can't find my comment. Mimir could also be an option.

[valyala]

Please consider also VictoriaLogs.

[arpanghosh8453]

Hi @valyala, can you please outline the difference of this from VictoriaMetrics?

Edit : I see you are working for Victoriametrics, so you would be able to describe it the best!

[joryirving]

The current solution feels pretty heavy handed. Why not pivot into being an *-exporter where it scrapes Garmin data, and exposes it to be consumed by Prometheus/VictoriaMetrics. Prometheus uses it's own TSDB, and so does VictoriaMetrics.

Something like https://github.com/prometheus-community/smartctl_exporter, https://github.com/prometheus/blackbox_exporter, or even https://github.com/DRuggeri/nut_exporter

[dcelasun]

Agreed. It would simplify this repo a lot and make the data available to non-Grafana consumers.

[arpanghosh8453]

Hello peeps! I dropped a new project for importing and visualizing Garmin FIT file data : https://github.com/arpanghosh8453/fit-dashboard!

I am sure you guys will like it :) - it's completely offline processing of FIT files in a modern interface. It uses duckDB for storage.

[depau]

It looks very nice, good job!

[DaanHend]

Thanks for letting us know. Looks fancy!

Any plans on further features?

[arpanghosh8453]

@DaanHend I will see if there are some interest in that app, and if so I will give it more attention depending on how people like and want to use it.

[arpanghosh8453]

@DaanHend and @depau let me know your experience with it :)

[joryirving]

Any way to use a prometheus backend?

[yihong0618]

any chance to give a try for greptimedb? https://github.com/GreptimeTeam/greptimedb
I did some work for it, its a quite fast and easy to use time series DB

cc @sunng87