Using multiple DD versions in the same environment¶

Whereas the default IMAS High Level Interface is built for a single Data Dictionary version, IMAS-Python can transparently handle multiple DD versions.

By default, IMAS-Python uses the same Data Dictionary version as the loaded IMAS environment is using, as specified by the environment variable IMAS_VERSION. If no IMAS environment is loaded, the last available DD version is used.

You can also explicitly specify which IMAS version you want to use when constructing a DBEntry or IDSFactory. For example:

import imas

factory_default = imas.IDSFactory()  # Use default DD version
factory_3_32_0 = imas.IDSFactory("3.32.0")  # Use DD version 3.32.0

# Will write IDSs to the backend in DD version 3.32.0
dbentry = imas.DBEntry("imas:hdf5?path=dd3.32.0-output/", "w", dd_version="3.32.0")
dbentry.create()

Conversion of IDSs between DD versions¶

IMAS-Python can convert IDSs between different versions of the data dictionary. This uses the “non-backwards compatible changes” metadata from the DD definitions. There are two conversion modes:

1. Automatic conversion: this is handled when reading or writing data (get()/get_slice(), put()/put_slice()).

   The DBEntry class automatically converts IDSs to the requested version:

   • When doing a put or put_slice, the provided IDS is automatically converted to the target version of the DBEntry when putting to disk.

   • When doing a get or get_slice, the IDS is automatically converted from the data dictionary version it was stored in (by checking ids_properties/version_put/data_dictionary) to the requested target version.

   Caution

   The automatic conversion doesn’t provide feedback when data cannot be converted between two versions of the data dictionary. Any incompatibilities between versions are silently ignored.

2. Explicit conversion: this is achieved with a call to imas.convert_ids.

Automatic conversion is faster when reading data (up to a factor 2, depending on the backend and the stored data), but it doesn’t support all conversion logic (see Supported conversions).

Recommendations for reading data

• Use automatic conversion when converting IDSs between Data Dictionary versions that have the same major version, unless you require a feature that is not supported by the automatic conversion.

• Use explicit conversion (see the example below) when converting IDSs between different major versions of the Data Dictionary.

• If you’re often reading the same data from a different DD version, it may be more efficient to convert the data to your DD version, store it and then use it. This avoids conversion every time you read the data.

  Converting an entire Data Entry can also be done with the IMAS-Python command line interface. See IMAS-Python Command Line tool.

Explicit conversion¶

import imas

entry = imas.DBEntry("<URI to data>", "r")

# Disable automatic conversion when reading the IDS with autoconvert=False
ids = entry.get("<ids name>", autoconvert=False)
# Explicitly convert the IDS to the target version
ids = imas.convert_ids(ids, "<target DD version>")

import imas

# Create a pulse_schedule IDS in version 3.23.0
ps = imas.IDSFactory("3.25.0").new("pulse_schedule")
ps.ec.antenna.resize(1)
ps.ec.antenna[0].name = "IDS conversion test"

# Convert the IDS to version 3.30.0
ps330 = imas.convert_ids(ps, "3.30.0")
# ec.antenna was renamed to ec.launcher between 3.23.0 and 3.30.0
print(len(ps330.ec.launcher))  # 1
print(ps330.ec.launcher[0].name.value)  # IDS conversion test

DD3 -> DD4 special rule: name + identifier -> description + name (GH#59)

IMAS‑Python implements an additional explicit conversion rule (see GH#59) to improve migration of Machine Description parts of IDSs when moving from major version 3 to 4. The rule targets simple sibling pairs on the same parent that provide both a “name” and an “identifier” field and that are NOT part of an “identifier structure” (the parent must not also have an “index” sibling). When applicable the rule performs the following renames during explicit DD3->DD4 conversion:

• DD3: parent/name -> DD4: parent/description

• DD3: parent/identifier -> DD4: parent/name

The conversion is applied only when the corresponding target fields exist in the DD4 definition and when no earlier mapping already covers the same paths. This is performed by the explicit conversion machinery (for example via imas.convert_ids or DBEntry explicit conversion) and is not guaranteed to be applied by automatic conversion when reading/writing from a backend.

In some cases like the one above, reverse conversion is also allowed(DD 4.0.0 -> 3.41.1)

Supported conversions¶

The following table shows which conversions are supported by the automatic and explicit conversion mechanisms.

Loading IDSs from a different major version¶

If you try to load an IDS that was stored in a different major version of the DD than you are using, IMAS-Python will raise a runtime error, for example:

On-disk data is stored in DD 3.39.1 which has a different major version than the
requested DD version (4.0.0). IMAS-Python will not automatically convert this
data for you.

You need to explicitly convert the data, which you can do as follows:

# Opened data entry
entry = imas.DBEntry(...)

# A plain get, or get_slice will raise a RuntimeError when the data is stored in
# a different major version of the DD:
# entry.get("equilibrium")

# So instead, we'll load the IDS in the DD version it is stored on disk
tmp_eq = entry.get("equilibrium", autoconvert=False)
# And explicitly convert it to the target version
equilibrium = imas.convert_ids(tmp_eq, entry.dd_version)

Storing IDSs with a different major version¶

If you try to put an IDS that was created for a different major version of the DD than the Data Entry you want to store it in, IMAS-Python raise a runtime error, for example:

Provided IDS uses DD 3.42.2 which has a different major version than the Data
Entry (4.0.0). IMAS-Python will not automatically convert this data for you.

You need to explicitly convert the data, which you can do as follows:

# IDS with data, in DD 3.42.2
equilibrium = imas.IDSFactory("3.42.2").equilibrium()
...

# Data Entry uses DD 4.0.0
with imas.DBEntry(uri, "w", dd_version="4.0.0") as entry:
    # This put would raise a runtime error, because the major version of the IDS
    # and the DBEntry don't match:
    # entry.put(equilibrium)

    # So instead, we'll explicitly convert the IDS and put that one
    entry.put(imas.convert_ids(equilibrium, entry.dd_version))

UDA backend caching and Data Dictionary versions¶

If you try to load data from a different Data Dictionary version with the UDA backend, you may see the following error:

The Data Dictionary version of the data (3.38.1) is different from the Data
Dictionary version of the DBEntry (3.42.0). This is not supported when using the
UDA backend.

There are three possible workarounds. The first two require passing an additional option in the IMAS UDA URI: please see the imas-core documentation for more details on these URI options.

1. Use UDA fetch to bypass the cache problem. You can do this by appending &fetch=1 to the URI when you create the DBEntry.

   Note that this will download the entire IDS files from the remote server, this may not be desired if you only want to read a single time slice.

2. Disable the UDA cache. You can do this by appending &cache_mode=none to the URI when you create the DBEntry.

   Note that this may make the get() (a lot) slower, since a separate request needs to be sent to the remote UDA server for every data variable. However, this may still be the best performing option if you are only interested in a subset of all the data in an IDS (and use Lazy loading).

3. Explicitly provide the data dictionary version when you create the DBEntry, setting it to match the Data Dictionary version of the data you want to load. To obtain the version of the data on the remote server from the field ids_properties.put_version.data_dictionary via a _lazy_ get() with autoconvert=False option and using the &cache_mode=none query in the URI.

   Note that you may need to call imas.convert_ids to convert the IDS to your desired Data Dictionary version.

All three possible workarounds are shown in the examples below:

import imas

URI = (
    "imas://uda.iter.org:56565/uda?backend=hdf5"
    "&path=/work/imas/shared/imasdb/ITER/3/121013/50"
)
with imas.DBEntry(URI, "r") as entry:
    cp = entry.get("core_profiles")

import imas

URI = (
    "imas://uda.iter.org:56565/uda?backend=hdf5"
    "&path=/work/imas/shared/imasdb/ITER/3/121013/50&fetch=1"
)
with imas.DBEntry(URI, "r") as entry:
    cp = entry.get("core_profiles")

import imas

URI = (
    "imas://uda.iter.org:56565/uda?backend=hdf5"
    "&path=/work/imas/shared/imasdb/ITER/3/121013/50&cache_mode=none"
)
with imas.DBEntry(URI, "r") as entry:
    cp = entry.get("core_profiles")

import imas

URI = (
    "imas://uda.iter.org:56565/uda?backend=hdf5"
    "&path=/work/imas/shared/imasdb/ITER/3/121013/50"
)
with imas.DBEntry(URI, "r", dd_version="3.38.1") as entry:
    cp = entry.get("core_profiles")

# Optional: convert the IDS to your desired DD version
cp = imas.convert_ids(cp, "3.42.0")

Background information¶

Data Dictionary definitions¶

The Data Dictionary definitions used by IMAS-Python are provided by the IMAS Data Dictionaries package. Please update this package if you need a more recent version of the data dictionary. For example, using pip:

pip install --upgrade imas-data-dictionaries