arbiter-migrations-0.1.0.0
arbiter-migrations
Safe HaskellNone
LanguageGHC2024

Arbiter.Migrations

Description

Versioned, tracked migrations for job queue schemas, run once in order. History lives in a schema_migrations table inside the target schema. Each run also seeds the admission policies and reconciles the rate-limit durability and optional triggers in MigrationConfig.

Synopsis

Configuration

data MigrationConfigSource #

Migration options and the desired state reconciled after a schema's tracked migrations run.

Constructors

MigrationConfig 

Fields

  • enableNotifications :: Bool

    Install the LISTEN/NOTIFY triggers that wake workers. Each run adds or drops them to match. Default: True.

  • enableEventStreaming :: Bool

    Install the admin UI's event-streaming triggers, which send a JSON event via pg_notify for every insert, update and delete on a queue table and every insert on its DLQ. A lease extend sends no event. Each run adds or drops them to match, and drops the shared function when no trigger uses it. Default: False.

  • rateLimitDurability :: Durability

    WAL-logging for the rate-limit bucket table in this schema. Unlogged (default) resets buckets on crash/failover. Durable preserves them at a throughput cost.

  • migrationLockTimeout :: Maybe NominalDiffTime

    Seconds to wait for the schema's migration lock, then return MigrationError. Nothing (default) waits indefinitely.

defaultMigrationConfig :: MigrationConfigSource #

Notify triggers on, event streaming off, unlogged rate-limit buckets, no lock timeout.

Tracked migrations

runMigrationsForRegistry :: forall (registry :: JobPayloadRegistry). (RegistryAdmissionPolicies registry, RegistryTables registry) => Proxy registry -> ByteString -> SchemaName -> MigrationConfig -> IO (MigrationResult String)Source #

Arguments

:: forall (registry :: JobPayloadRegistry). (RegistryAdmissionPolicies registry, RegistryTables registry) 
=> Proxy registry

Proxy for the job payload registry

-> ByteString

Database connection string

-> SchemaName

Schema for the arbiter tables

-> MigrationConfig

Migration options

-> IO (MigrationResult String) 

Migrate every queue in a registry into one schema. The schema itself is created first, outside migration tracking. Names that fail validateRegistryNames return MigrationError before the connection opens.

type AppRegistry =
  '[ Queue "email_jobs" EmailPayload
   , Queue "order_jobs" OrderPayload
   ]

main :: IO ()
main = do
  result <- runMigrationsForRegistry
              (Proxy @AppRegistry)
              "host=localhost dbname=mydb"
              "arbiter"
              defaultMigrationConfig
  print result

runMigrationsTrackedForTables :: ByteString -> SchemaName -> [(TableName, TableAdmission)] -> MigrationConfig -> AdmissionSeeds -> IO (MigrationResult String)Source #

Arguments

:: ByteString

Database connection string

-> SchemaName

Schema for the arbiter tables

-> [(TableName, TableAdmission)]

Every queue table in the schema, with its admission trigger kinds

-> MigrationConfig

Migration options

-> AdmissionSeeds

Admission policy rows to seed

-> IO (MigrationResult String) 

Migrate several tables in one schema. On success, seed the policies, then reconcile durability and triggers on the same connection. The table list must be the schema's whole queue set. An omitted queue loses its notify and event-streaming objects. Names that fail validateRegistryNames return MigrationError before the connection opens.

jobQueueMigrationsForTable :: SchemaName -> TableName -> TableAdmission -> [MigrationCommand]Source #

Arguments

:: SchemaName

Schema for the arbiter tables

-> TableName

Queue table name

-> TableAdmission

Which admission trigger kinds to install

-> [MigrationCommand] 

One queue's tracked migrations, each under its own version identifier. The optional notify and event-streaming objects are reconciled after the migrations run.

schemaLevelMigrations :: SchemaName -> [MigrationCommand]Source #

The schema-level migrations, run once per schema. The optional notification and event-streaming objects are reconciled after the migrations run.

data AdmissionSeedsSource #

Admission policy rows to seed after a successful migration.

Constructors

AdmissionSeeds 

Fields

noAdmissionSeeds :: AdmissionSeedsSource #

Seeds for a deployment with no admission policies.

data TableAdmissionSource #

Which admission trigger kinds a table's payload declares. Trigger migrations are install-only. A kind removed from a payload keeps its triggers.

Constructors

TableAdmission 

Fields

Instances

Instances details
Eq TableAdmission Source # 
Instance details

Defined in Arbiter.Migrations

Show TableAdmission Source # 
Instance details

Defined in Arbiter.Migrations

allTableAdmission :: TableAdmissionSource #

Install every admission trigger kind.

Validation

validateRegistryNames :: SchemaName -> [TableName] -> Either Text ()Source #

Reject an empty or over-long schema name, and queue names that are empty, exceed maxQueueNameBytes, generate a schema-wide arbiter table, or generate the same object or channel name as another queue.

maxQueueNameBytes :: IntSource #

The longest queue name whose generated identifiers stay distinct after PostgreSQL truncates them to maxIdentifierBytes.

conflictingPolicyPrefixes :: [RateLimitPolicy] -> [Text]Source #

Rate-limit policy prefixes declared with more than one parameter set.

Re-exports

data QueueSpec#

A queue's table name, payload type, and handler result type. A type data constructor. It takes no promotion tick.

Instances

Instances details
RegistryTables ('[] :: [QueueSpec]) # 
Instance details

Defined in Arbiter.Core.QueueRegistry

Methods

registryTableNames :: Proxy ('[] :: [QueueSpec]) -> [Text]#

registryQueueKinds :: Proxy ('[] :: [QueueSpec]) -> [(Text, [Text])]#

RegistryPolicies ('[] :: [QueueSpec]) p # 
Instance details

Defined in Arbiter.Core.Admission

(HasKind (SpecPayload spec), KnownSymbol (SpecName spec), NotInPayloads (SpecPayload spec) rest, NotInTables (SpecName spec) rest, RegistryTables rest) => RegistryTables (spec ': rest) # 
Instance details

Defined in Arbiter.Core.QueueRegistry

Methods

registryTableNames :: Proxy (spec ': rest) -> [Text]#

registryQueueKinds :: Proxy (spec ': rest) -> [(Text, [Text])]#

(CollectFor (SpecPayload spec) p, KnownSymbol (SpecName spec), RegistryPolicies rest p) => RegistryPolicies (spec ': rest) p # 
Instance details

Defined in Arbiter.Core.Admission

type Queue (table :: Symbol) payload = 'QueueWithResult table payload ()#

A queue whose handlers store no result. A module with its own Queue type needs import Arbiter.Core hiding (Queue) or a qualified import.

data MigrationCommand#

MigrationCommand determines the action of the runMigration script.

Constructors

MigrationInitialization

Initializes the database with a helper table containing meta information.

MigrationDirectory FilePath

Executes migrations based on SQL scripts in the provided FilePath in alphabetical order.

MigrationFile ScriptName FilePath

Executes a migration based on script located at the provided FilePath.

MigrationScript ScriptName ByteString

Executes a migration based on the provided bytestring.

MigrationValidation MigrationCommand

Validates that the provided MigrationCommand has been executed.

MigrationCommands [MigrationCommand]

Performs a series of MigrationCommands in sequence.

Instances

Instances details
Monoid MigrationCommand # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Semigroup MigrationCommand # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Eq MigrationCommand # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Ord MigrationCommand # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Read MigrationCommand # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Show MigrationCommand # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

data MigrationResult a#

A sum-type denoting the result of a migration.

Constructors

MigrationError a

There was an error in script migration.

MigrationSuccess

All scripts have been executed successfully.

Instances

Instances details
Functor MigrationResult # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Methods

fmap :: (a -> b) -> MigrationResult a -> MigrationResult b#

(<$) :: a -> MigrationResult b -> MigrationResult a#

Foldable MigrationResult # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Methods

fold :: Monoid m => MigrationResult m -> m#

foldMap :: Monoid m => (a -> m) -> MigrationResult a -> m#

foldMap' :: Monoid m => (a -> m) -> MigrationResult a -> m#

foldr :: (a -> b -> b) -> b -> MigrationResult a -> b#

foldr' :: (a -> b -> b) -> b -> MigrationResult a -> b#

foldl :: (b -> a -> b) -> b -> MigrationResult a -> b#

foldl' :: (b -> a -> b) -> b -> MigrationResult a -> b#

foldr1 :: (a -> a -> a) -> MigrationResult a -> a#

foldl1 :: (a -> a -> a) -> MigrationResult a -> a#

toList :: MigrationResult a -> [a]#

null :: MigrationResult a -> Bool#

length :: MigrationResult a -> Int#

elem :: Eq a => a -> MigrationResult a -> Bool#

maximum :: Ord a => MigrationResult a -> a#

minimum :: Ord a => MigrationResult a -> a#

sum :: Num a => MigrationResult a -> a#

product :: Num a => MigrationResult a -> a#

Traversable MigrationResult # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Methods

traverse :: Applicative f => (a -> f b) -> MigrationResult a -> f (MigrationResult b)#

sequenceA :: Applicative f => MigrationResult (f a) -> f (MigrationResult a)#

mapM :: Monad m => (a -> m b) -> MigrationResult a -> m (MigrationResult b)#

sequence :: Monad m => MigrationResult (m a) -> m (MigrationResult a)#

Eq a => Eq (MigrationResult a) # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Ord a => Ord (MigrationResult a) # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Read a => Read (MigrationResult a) # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

Show a => Show (MigrationResult a) # 
Instance details

Defined in Database.PostgreSQL.Simple.Migration

data Durability#

Whether the rate-limit bucket table is WAL-logged. Set by the migration config.

Constructors

Durable

WAL-logged. Buckets survive a crash.

Unlogged

Not WAL-logged. A crash resets the buckets. The migrations use this by default.

Instances

Instances details
Eq Durability # 
Instance details

Defined in Arbiter.Core.RateLimit.Spec

Show Durability # 
Instance details

Defined in Arbiter.Core.RateLimit.Spec

data RateLimitPolicy#

A token-bucket policy. Burst policyMax, refilling continuously at policyRefill per policyInterval. A policyRefill of 0 is a manually-refilled bucket. Fields are non-negative and the interval is positive. The migration rejects other values.

Constructors

RateLimitPolicy 

Fields

data ConcurrencyPolicy#

A concurrency policy. At most cpLimit jobs are in flight per key under cpPrefix. The default is seeded. An operator override on the policy takes precedence.

Constructors

ConcurrencyPolicy 

Fields

  • cpPrefix :: Text

    The key prefix. It must not contain :.

  • cpLimit :: Int32

    The default in-flight limit per key, in jobs. The migration rejects a declared limit below 1.

type SchemaName = Text#

PostgreSQL schema name, e.g. "arbiter".

type TableName = Text#

Unqualified table name within a schema, e.g. "email_jobs".