> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Adding data loading or unloading topics

This page provides a description of how to load a new file into a new
store within Atoti Market Risk. The techniques employed are generic
examples that can be extended, adapted and repeated for any use case you
encounter. In all cases, we make minimal changes to the MR application.

## Step 1 - Move data to the relevant directory

For this example, we want to bring any custom data to an appropriate
folder inside of : `/mr-application/src/main/resources/data` within the
`/2023-09-26` folder and name our file **Custom.csv**.

<table><thead><tr><th>AsOfDate</th><th>RiskClass</th><th>CustomProjected</th></tr></thead><tbody><tr><td>2018-09-26</td><td>Commodity</td><td>-6640.633693</td></tr><tr><td>2018-09-26</td><td>GIRR</td><td>14020.37649</td></tr><tr><td>2018-09-26</td><td>CSR Sec CTP</td><td>8386.767854</td></tr><tr><td>2018-09-26</td><td>CSR Sec non-CTP</td><td>19218.3336</td></tr><tr><td>2018-09-26</td><td>Commodity</td><td>-2460.048584</td></tr><tr><td>2018-09-26</td><td>Equity</td><td>8274.302903</td></tr><tr><td>2018-09-26</td><td>FX</td><td>2150.845785</td></tr><tr><td>2018-09-26</td><td>GIRR</td><td>17537.8908</td></tr><tr><td>2018-09-26</td><td>CSR non-Sec</td><td>-25353.39868</td></tr><tr><td>2018-09-26</td><td>DRC Sec non-CTP</td><td>11319.08548</td></tr><tr><td>2018-09-26</td><td>FX</td><td>25977.18728</td></tr><tr><td>2018-09-26</td><td>Commodity</td><td>11714.89133</td></tr><tr><td>2018-09-26</td><td>Equity</td><td>-19844.11309</td></tr><tr><td>2018-09-26</td><td>FX</td><td>8906.302165</td></tr><tr><td>2018-09-26</td><td>GIRR</td><td>19617.16455</td></tr><tr><td>2018-09-26</td><td>DRC non-Sec</td><td>17134.37517</td></tr></tbody></table>

## Step 2 - Define the datastore and references

To add a new datastore, we will need to create a `DatastoreConfiguratorConsumer` bean that will be used by
the `addModifications()` method inside the `DatastoreConfiguratorSetup` class.

```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
@Configuration
public class CustomDatastoreConfig {
    @Bean
    public DatastoreConfiguratorConsumer myCustomisations() {
        return Customisations::loadCustomisations;
    }

    public static void loadCustomisations(IDatastoreConfigurator configurator) {
        configurator.addStore(
                configurator.storeBuilder()
                        .withStoreName("CustomStore")
                        .withField(StoreFieldConstants.TRADE_KEY).asKeyField()
                        .withField("CustomField")
                        .withField("CustomValueField")
                        .build()
        );
        // Here we can also add a reference, if needed.
        configurator.addReference(
                configurator.referenceBuilder()
                        .fromStore(StoreConstants.TRADE_SENSITIVITIES_STORE_NAME)
                        .toStore("CustomStore")
                        .withName(DatastoreDescriptionUtils.joinName(StoreConstants.TRADE_SENSITIVITIES_STORE_NAME, "CustomStore"))
                        .withMapping(StoreFieldConstants.TRADE_KEY, StoreFieldConstants.TRADE_KEY)
                        .build()
        );
    }
}
```

See
[DatastoreHelper](https://docs.activeviam.com/products/tools/dash/3.4.0-AS6.1/online-help/)
for more information.

## Step 3 - Create channel parameters for our data loading topic

The channel parameters for our store include store name, topic name, and
file pattern.

Ensure that a file pattern property is available in the application, either explicitly in a file or as a default in a Spring Boot `ConfigurationProperties` object.

```
`mr.common.file-patterns.my-custom-file=**/*Custom*.csv`
```

The topic for the source can then be defined within a self-contained configuration class, by publishing a `ChannelParametersHolderOperator` bean:

```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
public class MyCustomSourceConfig {
    public static final String CUSTOM_STORE = "CustomStore";
    public static final List<String> ALIASES = List.of(DAILY_SENSI_ALIASES, ANY_TOPIC_ALIAS_NAME);

    @Bean
    @Qualifier(SP_QUALIFIER__COMMON_TOPIC_TO_STORE_AND_FILE_MAP)
    @Order(0)
    public ChannelParametersHolderOperator customSourceParameters(CustomFilePatternProperties filePatterns) {
        return patterns -> patterns.addTopic(CUSTOM_STORE, CUSTOM_STORE, filePatterns.getCustomFilePattern(), ALIASES);
    }
}
```

For more extensive customization, please refer to [Configuring sources using Spring Beans](./configure-sources).

## Step 4 - Include your custom configuration class in your application

You can include your new configuration class using the usual mechanisms: Spring auto-configuration or component scanning, explicit imports.

If you want to import it into the standard Atoti Market Risk application, navigate to the `MarketRiskConfig` class (`mr-application/src/main/java/com/activeviam/mr/application/main/MarketRiskConfig.java`) and add the class to the `@Import` annotation:

```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
@Configuration
@Import(value = {
//...
        MyCustomSourceConfig.class
//...
})
public class MarketRiskConfig {
    //...
}
```

## When using DirectQuery

When using DirectQuery, we must perform [Step 2](#step-2---define-the-datastore-and-references) and configure our datastore correctly. Then we have to modify our remote database
to include the new Table/Fields. When doing so, we must ensure we follow the naming convention of our
[Name Mapper](../dev-direct-query/customization-and-internals/name-mapper).

Our database only needs to contain the new fields and/or stores. When the application starts, the Datastore description is converted into a database
description. The remote database needs to contain the modifications before we start the application.

### Adding a Field with DirectQuery

When adding a new field, we must add it to the datastore description as outlined in [step 2](#step-2---define-the-datastore-and-references) and also ensure the field exists in our
database in the correct table.

Then when the application starts, the datastore description (containing the custom field) will be converted into a DirectQuery database description and the field
can be used for Hierarchies or treated the same as other DirectQuery fields.

<Note>
  When using custom fields with DirectQuery it is important to remember that datastore queries and getByKey queries are not recommended due to the performance
  overhead.
</Note>

### Adding a new Table with DirectQuery

Stores that are part of the datastore star schema are automatically added as tables used with DirectQuery.
If the remote table needs to be cached in memory, see paragraph \[Registering the Table to Cache]\(/dev/dev-direct-query/caching-remote-table.html#Registering the Table to Cache)

<Warning>
  The DirectQuery and in-memory Atoti servers must be configured identically. There must not be any hierarchies in the DirectQuery data node that do not exist in the in-memory one.
</Warning>

### Suggested Further Reading

* [Adding cube hierarchies](./add-a-new-cube-hierarchy)
* [Configuring measures using Spring Beans](./configure-measures)
* [Configuring schema selections using Spring Beans](./configure-schema-selections)
* [Configuring sources using Spring Beans](./configure-sources)
* [Adding a new KPI](./add-a-new-kpi)