Pre-processing

First steps required to set up a simulation

Mesh generation and requirements


As with all CFD solvers, defining a suitable mesh is one of the most important steps. XCALibre.jl does not provides mesh generation utilities (yet). Thus, the user will have to generate the grids using external mesh generation tools. However, XCALibre.jl does provide a number of mesh conversion tools to allow user to import their grids.

XCALibre.jl is an unstructured Finite Volume Method (FVM) library, therefore, we are able to support grids of arbitrary polyhedral cells. In XCALibre.jl a cell-centred FVM approach has been implemented, which is popular since it allows the representation of complex geometries and it is used by most commercial and many open source CFD solvers.

Units

XCALIbre.jl does not enforce units strictly. Units are implicitly provided by the mesh. You can use the scale keyword when importing a mesh to control the units used e.g. if the mesh file was created in millimetres, set scale=0.001 to work in metres. Thus, it is important to ensure that consistent units are used through all configuration entries. It is highly recommended that SI units are used.

Mesh conversion

XCALibre.jl at present supports .unv mesh formats (which can be generated using SALOME) for simulations in 2D and 3D domains. XCALibre.jl also supports the OpenFOAM mesh format for simulations in 3D only (for now).

Note

Currently, XCALibre.jl only supports loading mesh files stored in ASCII format. Please ensure that when saving grid files they are not saved in binary format. Most mesh generation programmes offer the option to export in ASCII (text-based) formats.

The following functions are provided for importing mesh files:

XCALibre.UNV2.UNV2D_mesh — Function
UNV2D_mesh(meshFile; scale=1, integer_type=Int32, float_type=Float64)

Read and convert 2D UNV mesh file into XCALibre.jl

Input

  • meshFile – path to mesh file.

Optional arguments

  • scale – used to scale mesh file e.g. scale=0.001 will convert mesh from mm to metres defaults to 1 i.e. no scaling

  • integer_type - integer type of the mesh indices; Int64 is needed only when a mesh has more than 2^31 faces, face-node entries or matrix entries, and reading such a mesh as Int32 stops with an error saying so

  • float_type - select interger type to use in the mesh (Float32 may be useful on GPU runs)

source
XCALibre.UNV3.UNV3D_mesh — Function
UNV3D_mesh(unv_mesh::String; scale::Real=1.0, integer_type::Type=Int32, float_type::Type=Float64) -> Mesh3

Constructs a Mesh3 object from a Universal (UNV) file format.

Arguments

  • unv_mesh::String: The file path to the UNV mesh file.

Keyword Arguments

  • scale::Real=1.0: A scaling factor applied to the nodal coordinates.
  • integer_type::Type=Int32: The integer type used for topological indices and connectivity arrays; Int64 is needed only when a mesh has more than 2^31 faces, face-node entries or matrix entries, and reading such a mesh as Int32 stops with an error saying so.
  • float_type::Type=Float64: The floating-point type used for coordinates and geometric properties.

Returns

  • Mesh3: A fully constructed 3D mesh object containing nodes, cells, faces, boundaries, and populated geometric properties.
source
XCALibre.FoamMesh.FOAM3D_mesh — Function
FOAM3D_mesh(mesh_file; scale=1, integer_type=Int32, float_type=Float64)

Read and convert 3D OpenFOAM mesh file into XCALibre.jl. Note that, at present, it is not recommended to run 2D cases using meshes imported using this function.

Input

  • mesh_file – path to mesh file.

Optional arguments

  • scale – used to scale mesh file e.g. scale=0.001 will convert mesh from mm to metres defaults to 1 i.e. no scaling

  • integer_type - integer type of the mesh indices; Int64 is needed only when a mesh has more than 2^31 faces, face-node entries or matrix entries, and reading such a mesh as Int32 stops with an error saying so

  • float_type - select interger type to use in the mesh (Float32 may be useful on GPU runs)

source

These conversion functions will read mesh information and generate a mesh object (Mesh2 or Mesh3 depending on whether the mesh is 2D or 3D, respectively) with the following properties:

XCALibre.Mesh.Mesh3 — Type
struct Mesh3{VV, VTF, VR, VO, VI, VS, VB} <: AbstractMesh

3D unstructured mesh. Mesh2 has identical fields. Each cell, face and node property is stored as its own array; mesh.cells, mesh.faces and mesh.nodes return views that index as Cell, Face3D and Node elements. Boundary faces come first in the face arrays.

    cell_centre::VV      # cell centroid coordinates
    cell_volume::VTF     # cell volumes
    cell_nodes_range::VR # range of each cell's nodes in cell_nodes
    cell_faces_range::VR # range of each cell's faces in cell_faces, cell_neighbours, cell_nsign
    cell_nodes::VI       # node IDs of each cell
    cell_faces::VI       # internal face IDs of each cell
    cell_neighbours::VI  # neighbour cell IDs across each cell face
    cell_nsign::VS       # face normal sign per cell face (1 or -1)
    face_nodes_range::VR # range of each face's nodes in face_nodes
    face_ownerCells::VO  # owner cell ID pair of each face (equal on boundary faces)
    face_centre::VV      # face centre coordinates
    face_normal::VV      # face unit normals
    face_e::VV           # unit vectors between owner cell centres
    face_area::VTF       # face areas
    face_delta::VTF      # distance between owner cell centres (boundary: cell centre to nearest face point)
    face_weight::VTF     # linear interpolation weights
    face_nodes::VI       # node IDs of each face
    face_gDiff::VTF      # Laplacian face coefficient (derived, see `_gDiff`)
    boundaries::VB       # boundary patches
    node_coords::VV      # node coordinates
    node_cells_range::VR # range of each node's cells in node_cells
    node_cells::VI       # cell IDs around each node
    get_float::VTF       # one-element array tagging the mesh float type
    get_int::VI          # one-element array tagging the mesh integer type
    boundary_cellsID::VI # owner cell ID of each boundary face
    boundary_cells::VI   # owner cells of boundary faces, each once (derived)
    boundary_cell_faces_range::VR # range of each boundary cell's faces in boundary_cell_faces
    boundary_cell_faces::VI # boundary face IDs grouped by owner cell, ascending within a cell

Kernels that add boundary-face contributions to cells loop over boundary_cells and, for each, over its faces in boundary_cell_faces, so every cell sums its faces in one fixed order: results do not depend on the thread count or the scheduling.

source

Mesh reordering

Every kernel that loops over cells reads the values of each cell's neighbours, so its speed depends on how close those neighbours sit in memory. Some mesh generators number cells in an order that scatters neighbours across the whole mesh (unstructured tetrahedral meshes in particular), which leaves CPU caches and GPU memory transactions mostly unused. reorder_mesh! renumbers the cells, faces and nodes of a loaded mesh so that neighbours sit close together; the geometry and boundary patches are unchanged, and a mesh that is already well ordered is left as it is. Meshes are read in the numbering of their files; reordering is applied only when requested:

mesh = reorder_mesh!(UNV3D_mesh(mesh_file, scale=0.001))
XCALibre.Mesh.reorder_mesh! — Function
reorder_mesh!(mesh::Union{Mesh2,Mesh3}; method=:rcm) -> mesh

Reorder the cells, faces and nodes of a host mesh in place so that neighbouring cells sit close in memory. Cell loops read the values of each cell's neighbours, so meshes whose generator numbers cells without regard to adjacency (typical of unstructured tetrahedral meshes) run faster once reordered. Geometry, face orientation and boundary patches are unchanged; only the numbering differs. The extra memory is a few index vectors of the mesh's integer type and one buffer the size of the largest connectivity list.

  • method=:rcm: reverse Cuthill-McKee ordering of the cell graph (bounded index distance between neighbours).
  • method=:morton: Z-order curve of the cell centres (compact groups of consecutive cells).

Internal faces follow their lowest-numbered cell, boundary faces their owner cell within each patch, and nodes the order in which the cells first reach them. The mesh is left unchanged unless the new order shortens the mean index distance between neighbouring cells by at least 10%.

Only the mesh in memory changes; mesh files are not touched. Fields built on the reordered mesh, and results written from them, are in the new order: VTK output carries the mesh in every file, and OpenFOAM output replaces a constant/polyMesh numbered differently before writing any field (in a distributed run, distribute and partition_mesh do this when they partition the mesh). Serial meshes only; call before adapt, and before distribute or partition_mesh to reorder a mesh for a distributed run.

Example

mesh = reorder_mesh!(UNV3D_mesh("mesh.unv", scale=0.001))
source

Mesh limitations and requirements

In this section we summarise the key limitations of the mesh loaders presented above, and we also highlight specific requirements.

UNV mesh files

  • Only ASCII files are supported
  • For 2D simulations the mesh must be contained in the X-Y plane only
  • In 3D only hex, tet and prism elements are supported

OpenFOAM mesh files

  • Only ASCII files are supported
  • Boundary groups (inGroups) are read and ignored, so they do not need to be deleted manually
  • Boundary information is not preserved (walls, symmetry, etc)
  • 2D setups are not currently supported (but will be)

Backend selection


In XCALibre.jl the mesh object is very important, as it will not only provide geometry information about the simulation/s, but it is also used to automatically dispatch methods to run on the appropriate backend. Therefore, users must first select the backend they wish to use for the simulations, and then "adapt" the mesh to use the correct backend.

XCALIbre.jl aims to work with all the backends supported by KernelAbstractions.jl. However, since internally XCALibre.jl uses sparse arrays to reduce its memory footprint, some GPU backends are not currently supported since this functionality is not yet available. Thus, currently only a subset of backends are supported:

  • CPU (multithreaded and tested)
  • NVidia GPUs (tested)
  • AMD GPUs (not tested - feedback welcome)

Selecting a given backend is straight-forward. The examples below show how to assign a backend (CPU or GPU) to the symbol backend and converting the mesh object to run a simulation on the corresponding backend. The converted mesh is assigned to the symbol mesh_dev for clarity.

CPU backend

Selecting the CPU backend is straight-forward. See the example below. Notice that CPU() is a backend type provided by KernelAbstractions.jl which we re-export for convenience. Call activate_multithread with the backend before building the model: fields and equations are allocated with the backend it is given.

CPU Example

mesh = # call function to load mesh e.g. UNV2_mesh, UNV3_mesh or FOAM3D_mesh
backend = CPU()
activate_multithread(backend)
mesh_dev = mesh # dummy reference to emphasise the mesh in on our chosen dev (or backend)

Multi-socket CPUs: NUMA placement

On nodes with several NUMA domains (e.g. two-socket servers), memory is fastest when each thread works on data held in its own domain. Linux places a page in the domain of the thread that first writes it, but the mesh and the sparse matrices are built on a single thread, so without further steps all threads read them from one domain. Following the KernelAbstractions.jl advice, pin the threads and pass a static backend to activate_multithread: fields, equations and matrices are then allocated with it, so each thread first writes the chunk it later works on. Pass the mesh through first_touch before building the model to place it the same way:

using ThreadPinning
pinthreads(:cores)                              # threads must stay on their cores
mesh = # call function to load mesh e.g. UNV3_mesh or FOAM3D_mesh
backend = CPU(static=true)                      # chunk c of every loop runs on thread c
activate_multithread(backend)                   # fields and equations are built with this backend
mesh = first_touch(mesh)                        # place the mesh itself
mesh_dev = mesh

Results are unchanged; only where the memory lives changes. On a two-socket node with 16 NUMA domains this cut a 10M-cell steady RANS run on 96 threads by 31%; on a single-socket machine it makes no difference. Do not combine it with numactl --interleave, which overrides first-touch placement.

GPU backends

To execute the code on GPUS, the process is also quite simple, but does require a few additional steps.

  • Install the corresponding Julia library that supports your hardware. For NVidia GPUs, the CUDA.jl package is required. For AMD GPUs, the AMDGPU.jl package is needed.
  • Move the mesh object to the backend device using the adapt method, which for convenience we re-export from Adapt.jl

Example for Nvidia GPU

mesh = # call function to load mesh e.g. UNV2_mesh, UNV3_mesh or FOAM3D_mesh
backend = CUDABackend()
mesh_dev = adapt(backend, mesh) # make mesh object backend compatible and move to GPU

Example for AMD GPU

mesh = # call function to load mesh e.g. UNV2_mesh, UNV3_mesh or FOAM3D_mesh
backend = ROCBackend()
mesh_dev = adapt(backend, mesh) # make mesh object backend compatible and move to GPU

Hardware configuration


In order to configure the backend the Hardware function can be used.

XCALibre.Simulate.Hardware — Type
hardware = Hardware(backend, workgroup)

Struct used to configure the backend.

Inputs

  • backend: used to specify the backend e.g. CPU(), CUDABackend() or other backends supported by KernelAbstraction.jl
  • workgroup::Int this is an integer specifying the number of workers that cooperate in a parallel run. For GPUs this could be set to the size of the device's warp e.g. workgroup = 32. On CPUs, the default value in KernelAbstractions.jl is currently workgroup = 1024.

Output

This function returns a Hardware object with the fields backend and workgroup which are accessed by internally in XCALibre.jl to execute a given kernel in the target backend.

source