Runtime and solvers

Final steps before launching a simulation

Runtime setup


At this stage in the setup workflow, the user defines the runtime information to configure the runtime behaviour of the flow solver, including the time step to use (only meaningful for transient solutions), as well as information about how often to write results to disk. XCALibre.jl provides a the Runtime to perform this operation.

XCALibre.Solve.Runtime — Type
Runtime(; 
        # keyword arguments

        iterations::I, 
        write_interval::I, 
        time_step::N,
        adaptive::A
    ) where {I<:Integer,N<:Number} = begin
    
    # returned Runtime struct
    Runtime{I<:Integer,F<:AbstractFloat}
        (
            iterations=iterations, 
            dt=time_step, 
            write_interval=write_interval,
            adaptive=adaptive
        )
end

This is a convenience function to set the top-level runtime information. The inputs are all keyword arguments and provide basic information to flow solvers just before running a simulation.

Input arguments

  • iterations::Integer: specifies the number of iterations in a simulation run.
  • write_interval::Integer: defines how often simulation results are written to file (on the current working directory). The interval is currently based on number of iterations. Set to -1 to run without writing results to file.
  • time_step::AbstractFloat: the time step to use in the simulation. Notice that for steady solvers this is simply a counter and it is recommended to simply use 1.
  • adaptive::Union{Nothing, AdaptiveTimeStepping}: optionally enables adaptive time stepping. Pass an AdaptiveTimeStepping object to automatically adjust dt based on the Courant number during transient simulations. Defaults to nothing, meaning a fixed time step is used.

Example

runtime = Runtime(iterations=2000, time_step=1, write_interval=2000)
source

Configuration object


Once all the simulation configuration information has been defined, from discretisation scheme to runtime information, all settings must be wrapped in a Configuration object. The definition, including expected input arguments, for the Configuration object are detailed below.

XCALibre.Simulate.Configuration — Type
struct Configuration{SC,SL,RT,HW,PP}
    schemes::SC
    solvers::SL
    runtime::RT
    hardware::HW
    postprocess::PP
end

The Configuration type is passed to all flow solvers and provides all the relevant information to run a simulation.

Inputs

  • schemes::NamedTuple this keyword argument is used to pass distretisation scheme information to flow solvers. See Numerical setup for details.
  • solvers::NamedTuple this keyword argument is used to pass the configurations for the linear solvers for each field information to flow solvers. See Runtime and solvers for details.
  • runtime::NamedTuple this keyword argument is used to pass runtime information to the flow solvers. See Runtime and solvers for details.
  • hardware::NamedTuple this keyword argument is used to pass the hardware configuration and backend settings to the flow solvers. See Pre-processing for details.

Optional keywords

  • postprocess this keyword argument is used to pass any fields that need to be post-processed. See Post-processing for details.

Example

config = Configuration(
    solvers=solvers, schemes=schemes, runtime=runtime, hardware=hardware, boundaries=BCs)
source

Initialising fields


The last (optional) step before running the simulation is to provide an initial guess for all the fields being solved. Although this step is optional, in most cases the flow solvers will perform better when initialised. To set an initial value for a field, the initialise! function is provided, which assigns a starting value to a given field.

XCALibre.Fields.initialise! — Function
function initialise!(field, value) # dummy function for documentation
    # Assign `value` to field in-place
    nothing
end

This function will set the given field to the value provided in-place. Useful for initialising fields prior to running a simulation.

Input arguments

  • field specifies the field to be initialised. The field must be either a AbractScalarField or AbstractVectorField
  • value defines the value to be set. This should be a scalar or vector (3 components) depending on the field to be modified e.g. for an AbstractVectorField we can specify as value=[10,0,0]

Note: in most cases the fields to be modified are stored within a physics model i.e. a Physics object. Thus, the argument value must fully qualify the model. For example, if we have created a Physics model named mymodel to set the velocity field, U, we would set the argument field to mymodel.momentum.U. See the example below.

Example

initialise!(mymodel.momentum.U, [2.5, 0, 0])
initialise!(mymodel.momentum.p, 1.25)

initialise! also accepts a functions arguments and initialises a field by evaluating the function at each cell centre. The function should have the signature f(x, y, z) and it must return either a scalar or an SVector type (requirement to work on GPUs)

source

A uniform initial guess leaves a velocity field that does not satisfy continuity, which the first few solver iterations must work off before making physical progress. For external and internal flows around bodies, a potential-flow field is a much better starting point: it already satisfies continuity and carries the shape of the geometry.

potential_flow! solves a velocity-potential equation on the current mesh and projects the velocity field onto the resulting divergence-free field. Boundary conditions for the potential are inferred from those already assigned to pressure, so no extra setup is needed.

XCALibre.Solvers.potential_flow! — Function
potential_flow!(model, config; ncorrectors=0, pref=nothing, time=0)

Project the current velocity field onto a divergence-free potential-flow field. Velocity boundary conditions supply the initial face flux. Velocity-potential boundary conditions are inferred from pressure: fixed pressure becomes fixed zero potential, periodic patches remain periodic, and other patches use zero normal gradient.

The corrected face-volume flux is returned with the linear-solver residual.

source

Call it after initialise! and before run!:

initialise!(model.momentum.U, velocity)
initialise!(model.momentum.p, 0.0)

potential_flow!(model, config; ncorrectors=5)

residuals = run!(model, config)

On meshes with appreciable non-orthogonality, pass ncorrectors to run non-orthogonal correctors on the potential equation. Set it to zero for orthogonal meshes.

AMG solver


The AMG linear solver can be selected directly in SolverSetup. It supports mode=AMGSolver() for a standalone multigrid solve and mode=Cg() for AMG-preconditioned conjugate gradient on symmetric systems such as pressure equations. mode takes an instance, not a symbol: mode=:cg throws an ArgumentError.

SolverSetup(
    solver = AMG(
        mode = Cg(),
        coarsening = SmoothAggregation(),
        smoother = AMGJacobi()
    ),
    preconditioner = Jacobi(),
    convergence = 1e-7,
    relax = 1.0,
    rtol = 0.0,
    atol = 1e-5
)

For non-symmetric systems use mode=Bicgstab(), which gives AMG-preconditioned stabilised biconjugate gradient. A typical case is the pressure equation in compressible flow, where the implicit pressure convection is upwinded and therefore non-symmetric. Cg() rejects a non-symmetric matrix. examples/2D_cylinder_transonic_RANS_AMG_BICGStab.jl uses it for the pressure equation of a transonic RANS case.

Choice of coarsening for non-symmetric operators: Bicgstab() has been tested with the default SmoothAggregation(). RugeStuben() and Geometric() are accepted but have not been validated on non-symmetric operators - RugeStuben() builds its strength of connection from each row's entries only, not from the transpose - so prefer the default unless you have checked the alternative on your case.

If Bicgstab() breaks down - the shadow residual going orthogonal, or the stabilising step collapsing - the half-step iterate is kept and the solve restarts with a fresh shadow residual, at most twice.

scale_correction (on by default) is supported. It makes the V-cycle preconditioner depend on its input, but the solver is right-preconditioned, so the solution and residual it carries stay consistent, and convergence is confirmed against the true residual b - A*x.

SolverSetup(
    solver = AMG(
        mode = Bicgstab(),
        smoother = AMGGaussSeidel(sweep = AMGForwardSweep())
        ),
    preconditioner=DILU(),
    convergence = 1e-7,
    relax = 1.0,
    rtol = 0.0,
    atol = 1e-5     
)

Solving the coarsest level

The coarsest level of the hierarchy is small but is visited on every V-cycle, so on a GPU the default host round trip can dominate. coarse_solve selects how it is solved:

  • OnDevice(; max_rows = 512) (default) — device-resident factorisation, no host round trip. A coarsest level with more rows than max_rows falls back to the host, since the factor is rebuilt on every coarse refresh.
  • OnHost() — always copy the coarsest right-hand side to the host and solve there. Also written coarse_solve = CPU().
  • OnDeviceJacobi(; omega, iterations) and OnDeviceChebyshev(; degree, eig_ratio, lambda_scale) — apply a fixed number of smoother sweeps on the device instead of solving. Both are constant linear operators, so they are valid with mode = Cg() as well as mode = AMGSolver().
  • OnDeviceKrylov(; solver, rtol, atol, itmax) — run an inner Krylov solve on the device. An inner Krylov solve is nonlinear in its right-hand side, which breaks the fixed-operator assumption of preconditioned CG, so this one requires mode = AMGSolver() and is rejected with mode = Cg(). Passing a GPU backend, as in coarse_solve = CUDABackend(), selects it.
AMG(mode = AMGSolver(), coarsening = SmoothAggregation(), smoother = AMGJacobi(),
    coarse_solve = OnDeviceChebyshev(degree = 8))

On a CPU backend every OnDevice* option behaves as OnHost().

Launching flow solvers


In XCALibre.jl the run! function is used to start a simulation, which will dispatch to the appropriate flow solver for execution. Once the simulation is complete a NamedTuple containing residual information is returned for users to explore the convergence history of the simulation.

XCALibre.Solvers.run! — Method
function run!(
    model::Physics, config; 
    output=VTK(), pref=nothing, ncorrectors=0, inner_loops=0
    )

    # here an internal function is used for solver dispatch
    return residuals
end

This is the top level API function to initiate a simulation. It uses the user-provided model defined as a Physics object to dispatch to the appropriate solver.

Dispatched flow solvers

  • Steady incompressible (SIMPLE algorithm for coupling)
  • Transient incompressible (PISO algorithm for coupling)
  • Steady weakly compressible (SIMPLE algorithm for coupling)
  • Transient weakly compressible (PISO algorithm for coupling)

Input arguments

  • model reference to a Physics model defined by the user.
  • config Configuration structure defined by the user with solvers, schemes, runtime and hardware structures configuration details.
  • output select the format used for simulation results from VTK() or OpenFOAM() (default = VTK())
  • pref Reference pressure value for cases that do not have a pressure defining BC. Incompressible solvers only (default = nothing)
  • ncorrectors number of non-orthogonality correction loops (default = 0)
  • inner_loops number to inner loops used in transient solver based on PISO algorithm (default = 0)

Output

This function returns a NamedTuple for accessing the residuals (e.g. residuals.Ux). The fields available within the returned residuals tuple depend on the solver used. For example, for an incompressible solver, a x-momentum equation residual can be retrieved accessing the Ux field i.e. residuals.Ux. Look at reference guide for each dispatch method to find out which fields are available.

Example

residuals = run!(model, config) 

# to access the pressure residual

residuals.p 
source

Restarting simulations


It should be noted that when running a simulation with run!, the solution fields in the Physics model are mutated. Thus, running the simulation from the previous solution is simply a matter of reissuing the run! function. At present, this has the side effect of overwriting any existing solution files (.vtk or .vtu). Users must be aware of this behaviour.

In some cases, it may be desirable to solve a problem with a steady solver and use the solution to run transient simulations. This is possible using the change function.

XCALibre.ModelPhysics.change — Function
change(model::Physics, property, value) => updatedModel::Physics

A convenience function to change properties of an exisitng Physics model.

Input arguments

  • model::Physics a Physics model to modify
  • property is a symbol specifying the property to change
  • value is the new setting for the specified property

Output

This function return a new Physics object

Example

To change a model to run a transient simulation e.g. after converging in steady state

modelTransient = change(model, :time, Transient())
source