Pre-processing
First steps required to set up a simulation
Mesh generation and requirements
As with all CFD solvers, defining a suitable mesh is one of the most important steps. XCALibre.jl does not provides mesh generation utilities (yet). Thus, the user will have to generate the grids using external mesh generation tools. However, XCALibre.jl does provide a number of mesh conversion tools to allow user to import their grids.
XCALibre.jl is an unstructured Finite Volume Method (FVM) library, therefore, we are able to support grids of arbitrary polyhedral cells. In XCALibre.jl a cell-centred FVM approach has been implemented, which is popular since it allows the representation of complex geometries and it is used by most commercial and many open source CFD solvers.
XCALIbre.jl does not enforce units strictly. Units are implicitly provided by the mesh. You can use the scale keyword when importing a mesh to control the units used e.g. if the mesh file was created in millimetres, set scale=0.001 to work in metres. Thus, it is important to ensure that consistent units are used through all configuration entries. It is highly recommended that SI units are used.
Mesh conversion
XCALibre.jl at present supports .unv mesh formats (which can be generated using SALOME) for simulations in 2D and 3D domains. XCALibre.jl also supports the OpenFOAM mesh format for simulations in 3D only (for now).
Currently, XCALibre.jl only supports loading mesh files stored in ASCII format. Please ensure that when saving grid files they are not saved in binary format. Most mesh generation programmes offer the option to export in ASCII (text-based) formats.
The following functions are provided for importing mesh files:
XCALibre.UNV2.UNV2D_mesh — Function
UNV2D_mesh(meshFile; scale=1, integer_type=Int32, float_type=Float64)Read and convert 2D UNV mesh file into XCALibre.jl
Input
meshFile– path to mesh file.
Optional arguments
scale– used to scale mesh file e.g. scale=0.001 will convert mesh from mm to metres defaults to 1 i.e. no scalinginteger_type- integer type of the mesh indices;Int64is needed only when a mesh has more than 2^31 faces, face-node entries or matrix entries, and reading such a mesh asInt32stops with an error saying sofloat_type- select interger type to use in the mesh (Float32 may be useful on GPU runs)
XCALibre.UNV3.UNV3D_mesh — Function
UNV3D_mesh(unv_mesh::String; scale::Real=1.0, integer_type::Type=Int32, float_type::Type=Float64) -> Mesh3Constructs a Mesh3 object from a Universal (UNV) file format.
Arguments
unv_mesh::String: The file path to the UNV mesh file.
Keyword Arguments
scale::Real=1.0: A scaling factor applied to the nodal coordinates.integer_type::Type=Int32: The integer type used for topological indices and connectivity arrays;Int64is needed only when a mesh has more than 2^31 faces, face-node entries or matrix entries, and reading such a mesh asInt32stops with an error saying so.float_type::Type=Float64: The floating-point type used for coordinates and geometric properties.
Returns
Mesh3: A fully constructed 3D mesh object containing nodes, cells, faces, boundaries, and populated geometric properties.
XCALibre.FoamMesh.FOAM3D_mesh — Function
FOAM3D_mesh(mesh_file; scale=1, integer_type=Int32, float_type=Float64)Read and convert 3D OpenFOAM mesh file into XCALibre.jl. Note that, at present, it is not recommended to run 2D cases using meshes imported using this function.
Input
mesh_file– path to mesh file.
Optional arguments
scale– used to scale mesh file e.g. scale=0.001 will convert mesh from mm to metres defaults to 1 i.e. no scalinginteger_type- integer type of the mesh indices;Int64is needed only when a mesh has more than 2^31 faces, face-node entries or matrix entries, and reading such a mesh asInt32stops with an error saying sofloat_type- select interger type to use in the mesh (Float32 may be useful on GPU runs)
These conversion functions will read mesh information and generate a mesh object (Mesh2 or Mesh3 depending on whether the mesh is 2D or 3D, respectively) with the following properties:
XCALibre.Mesh.Mesh3 — Type
struct Mesh3{VV, VTF, VR, VO, VI, VS, VB} <: AbstractMesh3D unstructured mesh. Mesh2 has identical fields. Each cell, face and node property is stored as its own array; mesh.cells, mesh.faces and mesh.nodes return views that index as Cell, Face3D and Node elements. Boundary faces come first in the face arrays.
cell_centre::VV # cell centroid coordinates
cell_volume::VTF # cell volumes
cell_nodes_range::VR # range of each cell's nodes in cell_nodes
cell_faces_range::VR # range of each cell's faces in cell_faces, cell_neighbours, cell_nsign
cell_nodes::VI # node IDs of each cell
cell_faces::VI # internal face IDs of each cell
cell_neighbours::VI # neighbour cell IDs across each cell face
cell_nsign::VS # face normal sign per cell face (1 or -1)
face_nodes_range::VR # range of each face's nodes in face_nodes
face_ownerCells::VO # owner cell ID pair of each face (equal on boundary faces)
face_centre::VV # face centre coordinates
face_normal::VV # face unit normals
face_e::VV # unit vectors between owner cell centres
face_area::VTF # face areas
face_delta::VTF # distance between owner cell centres (boundary: cell centre to nearest face point)
face_weight::VTF # linear interpolation weights
face_nodes::VI # node IDs of each face
face_gDiff::VTF # Laplacian face coefficient (derived, see `_gDiff`)
boundaries::VB # boundary patches
node_coords::VV # node coordinates
node_cells_range::VR # range of each node's cells in node_cells
node_cells::VI # cell IDs around each node
get_float::VTF # one-element array tagging the mesh float type
get_int::VI # one-element array tagging the mesh integer type
boundary_cellsID::VI # owner cell ID of each boundary face
boundary_cells::VI # owner cells of boundary faces, each once (derived)
boundary_cell_faces_range::VR # range of each boundary cell's faces in boundary_cell_faces
boundary_cell_faces::VI # boundary face IDs grouped by owner cell, ascending within a cellKernels that add boundary-face contributions to cells loop over boundary_cells and, for each, over its faces in boundary_cell_faces, so every cell sums its faces in one fixed order: results do not depend on the thread count or the scheduling.
Mesh reordering
Every kernel that loops over cells reads the values of each cell's neighbours, so its speed depends on how close those neighbours sit in memory. Some mesh generators number cells in an order that scatters neighbours across the whole mesh (unstructured tetrahedral meshes in particular), which leaves CPU caches and GPU memory transactions mostly unused. reorder_mesh! renumbers the cells, faces and nodes of a loaded mesh so that neighbours sit close together; the geometry and boundary patches are unchanged, and a mesh that is already well ordered is left as it is. Meshes are read in the numbering of their files; reordering is applied only when requested:
mesh = reorder_mesh!(UNV3D_mesh(mesh_file, scale=0.001))XCALibre.Mesh.reorder_mesh! — Function
reorder_mesh!(mesh::Union{Mesh2,Mesh3}; method=:rcm) -> meshReorder the cells, faces and nodes of a host mesh in place so that neighbouring cells sit close in memory. Cell loops read the values of each cell's neighbours, so meshes whose generator numbers cells without regard to adjacency (typical of unstructured tetrahedral meshes) run faster once reordered. Geometry, face orientation and boundary patches are unchanged; only the numbering differs. The extra memory is a few index vectors of the mesh's integer type and one buffer the size of the largest connectivity list.
method=:rcm: reverse Cuthill-McKee ordering of the cell graph (bounded index distance between neighbours).method=:morton: Z-order curve of the cell centres (compact groups of consecutive cells).
Internal faces follow their lowest-numbered cell, boundary faces their owner cell within each patch, and nodes the order in which the cells first reach them. The mesh is left unchanged unless the new order shortens the mean index distance between neighbouring cells by at least 10%.
Only the mesh in memory changes; mesh files are not touched. Fields built on the reordered mesh, and results written from them, are in the new order: VTK output carries the mesh in every file, and OpenFOAM output replaces a constant/polyMesh numbered differently before writing any field (in a distributed run, distribute and partition_mesh do this when they partition the mesh). Serial meshes only; call before adapt, and before distribute or partition_mesh to reorder a mesh for a distributed run.
Example
mesh = reorder_mesh!(UNV3D_mesh("mesh.unv", scale=0.001))Mesh limitations and requirements
In this section we summarise the key limitations of the mesh loaders presented above, and we also highlight specific requirements.
UNV mesh files
- Only ASCII files are supported
- For 2D simulations the mesh must be contained in the X-Y plane only
- In 3D only hex, tet and prism elements are supported
OpenFOAM mesh files
- Only ASCII files are supported
- Boundary groups (
inGroups) are read and ignored, so they do not need to be deleted manually - Boundary information is not preserved (walls, symmetry, etc)
- 2D setups are not currently supported (but will be)
Backend selection
In XCALibre.jl the mesh object is very important, as it will not only provide geometry information about the simulation/s, but it is also used to automatically dispatch methods to run on the appropriate backend. Therefore, users must first select the backend they wish to use for the simulations, and then "adapt" the mesh to use the correct backend.
XCALIbre.jl aims to work with all the backends supported by KernelAbstractions.jl. However, since internally XCALibre.jl uses sparse arrays to reduce its memory footprint, some GPU backends are not currently supported since this functionality is not yet available. Thus, currently only a subset of backends are supported:
- CPU (multithreaded and tested)
- NVidia GPUs (tested)
- AMD GPUs (not tested - feedback welcome)
Selecting a given backend is straight-forward. The examples below show how to assign a backend (CPU or GPU) to the symbol backend and converting the mesh object to run a simulation on the corresponding backend. The converted mesh is assigned to the symbol mesh_dev for clarity.
CPU backend
Selecting the CPU backend is straight-forward. See the example below. Notice that CPU() is a backend type provided by KernelAbstractions.jl which we re-export for convenience. Call activate_multithread with the backend before building the model: fields and equations are allocated with the backend it is given.
CPU Example
mesh = # call function to load mesh e.g. UNV2_mesh, UNV3_mesh or FOAM3D_mesh
backend = CPU()
activate_multithread(backend)
mesh_dev = mesh # dummy reference to emphasise the mesh in on our chosen dev (or backend)Multi-socket CPUs: NUMA placement
On nodes with several NUMA domains (e.g. two-socket servers), memory is fastest when each thread works on data held in its own domain. Linux places a page in the domain of the thread that first writes it, but the mesh and the sparse matrices are built on a single thread, so without further steps all threads read them from one domain. Following the KernelAbstractions.jl advice, pin the threads and pass a static backend to activate_multithread: fields, equations and matrices are then allocated with it, so each thread first writes the chunk it later works on. Pass the mesh through first_touch before building the model to place it the same way:
using ThreadPinning
pinthreads(:cores) # threads must stay on their cores
mesh = # call function to load mesh e.g. UNV3_mesh or FOAM3D_mesh
backend = CPU(static=true) # chunk c of every loop runs on thread c
activate_multithread(backend) # fields and equations are built with this backend
mesh = first_touch(mesh) # place the mesh itself
mesh_dev = meshResults are unchanged; only where the memory lives changes. On a two-socket node with 16 NUMA domains this cut a 10M-cell steady RANS run on 96 threads by 31%; on a single-socket machine it makes no difference. Do not combine it with numactl --interleave, which overrides first-touch placement.
GPU backends
To execute the code on GPUS, the process is also quite simple, but does require a few additional steps.
- Install the corresponding Julia library that supports your hardware. For NVidia GPUs, the CUDA.jl package is required. For AMD GPUs, the AMDGPU.jl package is needed.
- Move the mesh object to the backend device using the
adaptmethod, which for convenience we re-export from Adapt.jl
Example for Nvidia GPU
mesh = # call function to load mesh e.g. UNV2_mesh, UNV3_mesh or FOAM3D_mesh
backend = CUDABackend()
mesh_dev = adapt(backend, mesh) # make mesh object backend compatible and move to GPUExample for AMD GPU
mesh = # call function to load mesh e.g. UNV2_mesh, UNV3_mesh or FOAM3D_mesh
backend = ROCBackend()
mesh_dev = adapt(backend, mesh) # make mesh object backend compatible and move to GPUHardware configuration
In order to configure the backend the Hardware function can be used.
XCALibre.Simulate.Hardware — Type
hardware = Hardware(backend, workgroup)Struct used to configure the backend.
Inputs
backend: used to specify the backend e.g.CPU(),CUDABackend()or other backends supported byKernelAbstraction.jlworkgroup::Intthis is an integer specifying the number of workers that cooperate in a parallel run. For GPUs this could be set to the size of the device's warp e.g.workgroup = 32. On CPUs, the default value inKernelAbstractions.jlis currentlyworkgroup = 1024.
Output
This function returns a Hardware object with the fields backend and workgroup which are accessed by internally in XCALibre.jl to execute a given kernel in the target backend.