Global/wakapi
1
0
Fork 0
mirror of https://github.com/muety/wakapi.git synced 2025-12-06 06:22:41 -08:00
Code Issues Packages Projects Releases Wiki Activity

fix: workaround for postgres timestamp issue (resolve #761)

Browse Source
This commit is contained in:
Ferdinand Mütsch
2025-03-26 21:47:57 +01:00
parent bc2096f411
commit 14fae4a3c8
4 changed files with 546 additions and 504 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
config/db.go
View File

@@ -2,13 +2,12 @@ package config
import (
"fmt"
"net/url"
"github.com/glebarez/sqlite"
"gorm.io/driver/mysql"
"gorm.io/driver/postgres"
"gorm.io/driver/sqlserver"
"gorm.io/gorm"
"net/url"
)
/*
@@ -18,12 +17,17 @@ A quick note to myself including some clarifications about time zones.
- From my understanding, MySQL server tz is only a fallback and can be ignored as long as a connection tz is specified
- All times are currently stored inside TIMESTAMP columns (alternatives would be DATETIME and BIGINT (plain Unix timestamps))
- TIMESTAMP columns, to my understanding, do not keep any time zone information, but only the very time they store
- Setting a `loc` parameter specifies what location parsed time.Time objects will be in, however, does not affect the session time zone setting (https://github.com/go-sql-driver/mysql#loc)
- I.e., when not setting `time_zone` in addition, the session time zone will probably default to the server time zone (UTC in case of Docker)
- Session time zone will result in conversions of inserted times from that time zone to UTC
- From my understanding, TIMESTAMP only stores a plain time value without tz information and then converts it only for retrieval to whatever tz is set for the session
- Setting a `loc` parameter specifies location parsed time.Time objects are interpreted in, however, does not affect the session time zone setting (https://github.com/go-sql-driver/mysql#loc)
- I.e., when not setting `time_zone` in addition, the session time zone will default to the SYSTEM time zone (UTC in case of Docker)
- Session time zone will result in conversions of inserted times from that time zone to UTC (but, again, we don't use that)
- TIMESTAMP only stores a plain time value without tz information and then converts it only for retrieval to whatever tz is set for the session
- E.g., when inserting '2021-04-27 08:26:07' with session tz set to Europe/Berlin and then viewing the database table with UTC tz will return '2021-04-27 06:26:07' instead
- Currently, no session tz is set (only loc), so the database server will assume it receives UTC. However, as no tz is set when retrieving the values either, they are also going to be returned just as is and as long as `loc=Local` is set properly, they are parsed in Go code with the correct time zone
- Loc is a parameter of the driver and used in conjunction with parseTime. It results in time.Time objects being converted to that respective zone before storage and converted back to it during retrieval
- E.g., when inserting '2021-04-27 08:26:07 +01:00' with loc set to UTC results in the actual database column to show '2021-04-27 07:26:07'
- A loc value of "Local" is effective the same as setting loc to the value of the TZ environment variable (or system default otherwise)
- Note that if loc is different from "Local", when using models.CustomTime, values are first received in loc zone and then converted to whatever Wakapi's current zone is (e.g. specified through TZ env.), see https://github.com/muety/wakapi/blob/bc2096f4117275d110a84f5b367aa8fdb4bd87ba/models/shared.go#L95.
- Currently, we don't set a session tz (only loc), so the database server will assume it receives dates in its system time. However, as no tz is set when retrieving the values either, they are also going to be returned just as is and as long as `loc=Local` is set properly, they are parsed in Go code with the correct time zone
- In other words, using loc causes the Go driver to take care of zones, while using a session timezone causes MySQL to take care of zones (?)
- As long as the Wakapi server always runs in the same time zone, it will always parse these dates the same way (i.e. as time.Local, Europe/Berlin in case of Wakapi.dev)
- Using TIMESTAMP columns would only become problematic when either data needs to be migrated to a Wakapi instance in a different tz or if two consumers in different tzs were reading and writing to the same table
- It is important to have same `time_zone` and `loc` parameters set when sending and receiving, no matter what it is (writing / reading in 'UTC' will yield same results as writing / reading in 'Europe/Berlin')
@@ -37,6 +41,14 @@ A quick note to myself including some clarifications about time zones.
- A request with `?from=2021-04-25` from California (PST / UTC-7) would ideally have to be translated into a database query like `from >= 2021-04-25T00:00:00+0900)`, assuming that Wakapi runs at CEST (UTC+2)
- This translation comes from either the user explicitly requesting with a specified tz (i.e. sending `from` as ISO8601 / RFC3999) or them having specified a tz in their profile
- Implicit intervals are tricky, too, as they are generated on the server, but still have to respect the user's tz, as `today` is different for a user in Cali and one in Karlsruhe
For Postgres, the story is even a different one.
- There are `timestamp` and `timestamptz` columns, whereby the former seems to behave very strangely.
- Apparently, when storing dates, they'll just chop off zone information entirely, while for retrieval, all dates are interpreted as UTC
- If Wakapi runs in CEST, '2025-03-25T14:00:00 +02:00' will end up as '2025-03-25T14:00:00' in the database and become '2025-03-25T16:00:00 +02:00' when retrieved back (at least in case of our annoying models.CustomTime, because of https://github.com/muety/wakapi/blob/bc2096f4117275d110a84f5b367aa8fdb4bd87ba/models/shared.go#L95)
- For `timestamptz`, columns don't actually store tz information either (https://github.com/jackc/pgx/issues/520#issuecomment-479692198), but at least allow for correct retrieval. The driver will return dates already in the application's TZ
- Here (https://github.com/go-gorm/gorm/issues/4834) is an interesting discussion on the issue
- According to https://www.cybertec-postgresql.com/en/time-zone-management-in-postgresql/, good practice is to always use timestamptz, leaving conversions to the database itself
*/
func (c *dbConfig) GetDialector() gorm.Dialector {
@@ -90,6 +102,9 @@ func postgresConnectionString(config *dbConfig) string {
sslmode = "require"
}
// note: passing a `timezone` param here doesn't seem to have any effect, neither with `timestamp`, not for `timestamptz` columns
// possibly related to https://github.com/go-gorm/postgres/issues/199 ?
return fmt.Sprintf("host=%s port=%d user=%s dbname=%s password=%s sslmode=%s",
config.Host,
config.Port,
@@ -105,9 +120,7 @@ func sqliteConnectionString(config *dbConfig) string {
}
func mssqlConnectionString(config *dbConfig) string {
query := url.Values{}
query.Add("database", config.Name)
if config.Ssl {

985
coverage/coverage.out
View File

File diff suppressed because it is too large Load Diff

29
models/shared.go
View File

@@ -5,6 +5,9 @@ import (
"encoding/json"
"errors"
"fmt"
"github.com/muety/wakapi/config"
"github.com/muety/wakapi/utils"
"gorm.io/driver/postgres"
"strconv"
"strings"
"time"
@@ -21,6 +24,11 @@ const (
PersistentIntervalKey = "wakapi_summary_interval"
)
var (
hacksInitialized bool
postgresTimezoneHack bool
)
type KeyStringValue struct {
Key string `gorm:"primary_key"`
Value string `gorm:"type:text"`
@@ -39,14 +47,15 @@ type KeyedInterval struct {
// CustomTime is a wrapper type around time.Time, mainly used for the purpose of transparently unmarshalling Python timestamps in the format <sec>.<nsec> (e.g. 1619335137.3324468)
type CustomTime time.Time
// sql server doesn't allow multiple columns with timestamp type in a column
// So we need to change the type to datetimeoffset
func (j CustomTime) GormDBDataType(db *gorm.DB, field *schema.Field) string {
t := "timestamp"
// sql server doesn't allow multiple columns with timestamp type in a column
// So we need to change the type to datetimeoffset
if db.Config.Dialector.Name() == (sqlserver.Dialector{}).Name() {
t = "datetimeoffset"
} else if db.Config.Dialector.Name() == (postgres.Dialector{}).Name() {
// TODO: migrate to timestamptz, see https://github.com/muety/wakapi/issues/761
}
if scale, ok := field.TagSettings["TIMESCALE"]; ok {
@@ -92,15 +101,23 @@ func (j *CustomTime) Scan(value interface{}) error {
return errors.New(fmt.Sprintf("unsupported type: %T", value))
}
t = time.Unix(0, (t.UnixNano()/int64(time.Millisecond))*int64(time.Millisecond)) // round to millisecond precision
// see https://github.com/muety/wakapi/issues/762
// -> "reinterpret" postgres dates (received as UTC) in local zone, assuming they had also originally been inserted as such
if !hacksInitialized {
postgresTimezoneHack = config.Get().Db.IsPostgres()
}
if postgresTimezoneHack {
t = utils.SetZone(t, time.Local)
}
t = t.In(time.Local).Round(time.Millisecond)
*j = CustomTime(t)
return nil
}
func (j CustomTime) Value() (driver.Value, error) {
t := time.Unix(0, j.T().UnixNano()/int64(time.Millisecond)*int64(time.Millisecond)) // round to millisecond precision
return t, nil
return j.T().Round(time.Millisecond), nil
}
func (j *CustomTime) Hash() (uint64, error) {

5
utils/date.go
View File

@@ -58,6 +58,11 @@ func LocalTZOffset() time.Duration {
return time.Duration(offset * int(time.Second))
}
// SetZone overwrites a date's timezone without reinterpreting at
func SetZone(t time.Time, loc *time.Location) time.Time {
return time.Date(t.Year(), t.Month(), t.Day(), t.Hour(), t.Minute(), t.Second(), t.Nanosecond(), loc)
}
func ParseWeekday(s string) time.Weekday {
switch strings.ToLower(s) {
case "mon", strings.ToLower(time.Monday.String()):