AcediaCore/docs/API/collections.md

Collections

All Acedia's collections store AcediaObjects. By taking advantage of boxing we can use them to store arbitrary types: both value types (native variables and structs) and reference types (AcediaObject and it's children).

Currently Acedia provides dynamic arrays (regular integer-indexed array) and associative arrays (collection of key-value pairs with quick access to values via AcediaObject keys). Using them is fairly straightforward, but, since they're dealing with objects, some explanation about their memory management is needed. Below we attempt to give a detailed description of everything you need to know to efficiently use Acedia's collections.

Usage examples

Dynamic arrays

Dynamic arrays can be created via either of _.collections.EmptyDynamicArray() / _.collections.NewDynamicArray() methods. _.collections.NewDynamicArray() takes an array<Acedia> argument and populates returned DynamicArray with it's items, while _.collections.EmptyDynamicArray() simply creates an empty DynamicArray.

They are similar to regular dynamic array<AcediaObject>s with several differences:

1. They're passed by reference, rather than by value (no additional copies are made when passing DynamicArray as an argument to a function or assigning it to another variable);
2. They have richer interface;
3. They automatically handle necessary object deallocations.

As an example to illustrate basic usage of DynamicArray let's create a trivial class that remembers players' nicknames:

class PlayerDB extends AcediaObject;

var private DynamicArray storage;

//  Constructor and destructor allow for memory management
protected function Constructor()
{
    storage = _.collections.EmptyDynamicArray();
}

protected function Finalizer()
{
    storage.FreeSelf();
    storage = none;
}

public function RegisterNick(Text newNickName)
{
    if (newNickName == none)            return;
    //  `Find` returns `-1` if object is not found
    if (storage.Find(newNickName) >= 0) return;
    storage.AddItem(newNickName);
}

public function IsRegisteredID(Text toCheck)
{
    return (storage.Find(toCheck) >= 0);
}

public function ForgetNick(Text toForget)
{
    //  This method removes all instances of `toForget` in `storage`;
    //  There's also an optional flag to only remove the first one.
    storage.RemoveItem(toForget);
}

What happens if we deallocate stored objects?

They will turn into none:

local Text item;
local DynamicArray storage;
storage = _.collections.EmptyDynamicArray();
item = _.text.FromString("example");
storage.AddItem(item);
//  Everything is as expected here
TEST_ExpectNotNone(item);
TEST_ExpectNotNone(storage.GetItem(0));
TEST_ExpectTrue(storage.GetItem(0) == item);

//  Now let's deallocate `item`
item.FreeSelf();
//  Suddenly things are different:
TEST_ExpectNotNone(item); // `item` deallocated, but not dereferenced
TEST_ExpectNone(storage.GetItem(0)); // but it is gone from the collection
TEST_ExpectFalse(storage.GetItem(0) == item);

Let's explain what's changed after deallocation:

1. Even though we've deallocated item, its reference still points at the Text object. This happens because object being deallocated is simply Acedia's flag for it and, from the Unreal Engine's point of view, item still exists and is being used;
2. storage.GetItem(0) no longer points at that Text object. Unlike a simple array<AcediaObject>, DynamicObject tracks status of its items and replaces their values with none when they're deallocated. This kind of cleanup is something we cannot do with simple FreeSelf() or even _.memory.Deallocate() for object stored in a regular array, but can for objects stored in Acedia's Collections.
3. Since our collection has forgotten about item after it was deallocated, storage.GetItem(0) == item will be false.

What happens if we remove an item from our DynamicArray collection?

By default nothing - stored items will continue to exist outside the collection. This is because by default DynamicArray (and AssociativeArray) is not responsible for deallocation of its items. But it can be made to.

Suppose that to avoid items disappearing from our collections, we put in their copies instead. For Text it can be accomplished with a simple Copy() method: storage.AddItem(item.Copy()). This creates a problem - storage, as we've just explained, won't actually deallocate this item if we simply remove it. We will have to do so manually to prevent memory leaks:

...
_.memory.Deallocate(storage.GetItem(i));
storage.RemoveIndex(i);

which isn't ideal.

To solve this problem we can add a copy of an item to our DynamicArray as a managed object: collections will consider themselves responsible for deallocation of objects marked as managed and will automatically clean them up. To add an item as managed we need to simply specify second argument for AddItem(, true) method:

local Text item;
local DynamicArray storage;
storage = _.collections.EmptyDynamicArray();
item = _.text.FromString("example");
storage.AddItem(item, true);
//  Here added item is still allocated
TEST_ExpectTrue(item.IsAllocated());
//  But after it's removed from `storage`...
storage.RemoveIndex(0);
//  ...it's automatically gets deallocated
TEST_ExpectFalse(item.IsAllocated());

Whether you would want your collection to auto-deallocate your items or not depends only on how you plan to use your collections.

NOTE: The same collection can technically contain both managed and unmanaged items, but it is best you avoid mixing these types of items.

Let's rewrite RegisterNick() method of PlayerDB to make it independent from whether Text objects passed to it are deallocated:

...
public function RegisterNick(Text newNickName)
{
    if (newNickName == none)            return;
    if (storage.Find(newNickName) >= 0) return;
    //  Store an independent, but managed copy,
    //  that will be gone along with the `storage`
    storage.AddItem(newNickName.Copy(), true);
}
...

IMPORTANT: While items added to collections aren't managed by default, it is a convention that if you return a collection from your function and make another piece of code responsible for it - that piece of code also becomes responsible for that collection's items (meaning that it must deallocate them when they are no longer needed). If you expect a different behavior - you must specify so in the function's description.

Associative arrays

IMPORTANT: It is assumed you've read previous section about DynamicArrays and its managed objects first.

Associative arrays allow to efficiently store and access AcediaObject values via AcediaObject keys by using hash map under the hood. While objects of any AcediaObject's subclass can be used as keys, the main reason for implementing associative arrays was to allow for Text keys and examples in this sections will focus on them specifically.

The basic interface is simple and can be demonstrated with this:

local AcediaObject      item;
local AssociativeArray  storage;
storage = _.collection.NewAssociativeArray();
//  Add some values
storage.SetItem(_.text.FromString("year"), _.ref.int(2021));
storage.SetItem(    _.text.FromString("comment"),
                    _.text.FromString("What year it is?"));
//  Then get them
item = storage.GetItem(_.text.FromString("year"));
TEST_ExpectTrue(IntRef(item).Get() == 2021);
item = storage.GetItem(_.text.FromString("comment"));
TEST_ExpectTrue(Text(item).ToString() == "What year it is?");

In above example we've created separate text instances (with the same contents) to store and retrieve items in AssociativeArray. However it is inefficient to each time create Text anew just to get an item:

1. It defeats the purpose of using Text over string, since (after initial creation cost) Text allows for a cheaper access to individual characters and also allows us to compute Text's hash only once, caching it for later use. But if we create Text object every time we want to access value in AssociativeArray we will only get more overhead without any benefits.
2. It leads to creation of useless objects, that we didn't even deallocate in the above example.

So it is recommended that, whenever possible, your class would define reusable Text constant that it would want to use as keys beforehand. If you want to implement a class that receives zed's data as an AssociativeArray and wants to buff its health, you can do the following:

class MyZedUpgrader extends AcediaObject;

var protected Text TMAX_HEALTH;

protected function StaticConstructor()
{
    default.TMAX_HEALTH = _.text.FromString("maxhealth");
}

public final function UpgradeMyZed(AssociativeArray zedData)
{
    local IntRef maxHealth;
    maxHealth = IntRef(AssociativeArray.GetItem(TMAX_HEALTH));
    maxHealth.Set(maxHealth.Get() * 2);
}

Text has more information about convenient ways to efficiently create Text constants. For example, in the above use case of upgrading zed's health it is acceptable to do this instead:

class MyZedUpgrader extends AcediaObject;

public final function UpgradeMyZed(AssociativeArray zedData)
{
    local IntRef maxHealth;
    maxHealth = IntRef(AssociativeArray.GetItem(P("maxhealth")));
    maxHealth.Set(maxHealth.Get() * 2);
}

Memory management and AssociativeArray

AssociativeArray supports the concept of managed objects in the same way as DynamicArrays: by default objects are not managed, but can be added as such when optional argument is used: AssociativeArray.GetItem(P("value"), someItem, true). We'll just note here that it's possible to remove a managed item from AssociativeArray without deallocating it with TakeItem()/TakeEntry() methods.

A question specific for AssociativeArrays is whether they deallocate their keys. And the answer is: they do not. AssociativeArray will not deallocate its keys, even if a managed value is recorded with them. This way one can use the same pre-allocated key in several different AssociativeArrays. If you do need to deallocate them, you will have to do it manually.

One good way to do so is to use TakeEntry(AcediaObject key) method that returns a struct Entry with both key and recorded value inside:

struct Entry
{
    //  Non-public fields are omitted
    var public      AcediaObject    key;
    var public      AcediaObject    value;
    var public      bool            managed;
};

This method also always removes stored value from AssociativeArray without deallocating it, even if it was managed, making you responsible for it.

Another way is to completely clear your collection along with any keys inside with Empty(true) method. This method recursively clears your collection (also making Empty() calls on any collections stored inside yours) and passing it true as a parameter makes it deallocate any key objects used in these collections.

In case of the opposite situation, where one deallocates an AcediaObject used as a key, AssociativeArray will automatically remove appropriate entry in its entirety. However this is only a contingency measure: you should never deallocate objects that are still used as keys in AssociativeArray. One of the negative consequences is that it'll screw up AssociativeArray's GetLength() results, making it possibly overestimate the amount of stored items (because there is no guarantee on when an entry with deallocated key will be detected and cleaned up).

Capacity

Acedia's AssociativeArray works like a hash table and needs to allocate sufficiently large dynamic array as a storage for its items. If you keep adding new items that storage will eventually become too small for hash table to work efficiently and we will have to reallocate and re-fill it. If you want to add a huge enough amount of items into your AssociativeArray, this process might be repeated several times. This is not ideal, since it means doing a lot of iteration, each taking noticeable time and increasing infinite loop counter (game will crash if it gets high enough). AssociativeArray allows you to set minimal capacity with SetMinimalCapacity() method to force it to pre-allocate enough space for the expected amount of items. Setting minimal capacity to the maximum amount of items you expect to store in the caller AssociativeArray will remove any need for reallocating the storage.

NOTE: AssociativeArray always allocates storage array with length of at least MINIMUM_SIZE = 50 and won't need any reallocations before you add at least MINIMUM_SIZE * MAXIMUM_DENSITY = 50 * 0.75 ~= 38 items, no matter the current minimal capacity (that can be checked with GetMinimalCapacity() method).

[Advanced] Associative arrays' keys

AssociativeArray allows to store AcediaObject values by AcediaObject keys. Ultimately, any AcediaObject can be used for either, but behavior of the AssociativeArray regarding its key depends on how key's IsEqual() and GetHashCode() methods are implemented.

IMPORTANT: Refresh your knowledge on how equality checks for Acedia's objects work, do not rely on intuition here.

For example Text's hash and equality is determined by its content:

local Text t1, t2;
t1 = _.text.FromString("Some random text");
t2 = _.text.FromString("Some random text");
//  All of these assertions are correct:
TEST_ExpectTrue(t1.IsEqual(t2));                        //  same content
TEST_ExpectTrue(t1.GetHashCode() == t2.GetHashCode());  //  same hashes
TEST_ExpectTrue(t1 != t2);                              //  different objects

Therefore, if you used one Text as a key, then you will be able to obtain its value with another Text that contains the same string. However MutableText's contents can change dynamically, so it cannot afford to base its equality and hash on its contents:

local MutableText t1, t2;
t1 = _.text.FromStringM("Some random text");
t2 = _.text.FromStringM("Some random text");
//  `IsEqual()` no longer compares contents;
//  Use `Compare()` instead.
TEST_ExpectFalse(t1.IsEqual(t2));
TEST_ExpectFalse(t1.GetHashCode() == t2.GetHashCode()); //  different hashes (most likely)
TEST_ExpectTrue(t1 != t2);                              //  different objects

MutableText can still be used as a key, but value stored with it will only be obtainable by providing the exact instance of MutableText, regardless of its contents:

local MutableText       t1, t2;
local AssociativeArray  storage;
storage = _.collection.NewAssociativeArray();
t1 = _.text.FromStringM("Some random text");
t2 = _.text.FromStringM("Some random text");
storage.SetItem(t1, _.text.FromString("Contents!"));
TEST_ExpectNone(storage.GetItem(t2));
TEST_ExpectNotNone(storage.GetItem(t1));

As far as base Acedia's classes go, only Text and boxed (immutable ones, not refs) values are a good fit to be used as contents-dependent keys.

Better accessors

While you can store simple values inside these arrays in a straightforward manner of storage.SetItem(_.text.FromString("year"), _.ref.int(2021)), it is not very convenient. Especially getting items from such arrays can be problematic, since that int can potentially be stored as both immutable IntBox or mutable IntRef.

To help with this problem Acedia's collections provide a bunch of convenience accessors for UnrealScript's built-in types. Let us start with getters GetBool(), GetByte(), GetInt(), GetFloat(), GetText() (since Acedia uses Text instead of string). These take index for DynamicArray or AcediaObject keys for AssociativeArray and return relevant type if they find either box or a ref of such type in the caller array. All of them, except Text, also allow you to provide default value as a second argument - this value will be used if neither box or ref for the desired type is found.

Then there's setter methods SetBool(), SetByte(), SetInt(), SetFloat() that take at least two parameters: index/key and value to store. They automatically create either box or ref object to wrap around passed primitive value and always store it as a managed item. Third, optional, bool parameter asRef allows you to decide whether passed value should be saved inside the array in an immutable box or in a mutable ref (default false is to save that primitive type in a box).

NOTE: There is no paired SetText() setter for GetText() getter, since Text itself is an object and can directly be saves with SetItem().

Here is an example of how they work:

local IntBox        box;
local IntRef        ref;
local DynamicArray  storage;
storage = _.collection.NewDynamicArray();
storage.SetInt(0, 7);
//  `int` value is not returned normally, but there is not auto-conversion
//  into `float` and so `GetFloat()` returns provided default value instead
Log("Value as int:" @ storage.GetInt(0));           //  Value as int: 7
Log("Value as float:" @ storage.GetFloat(0, 9));    //  Value as int: 9

box = IntBox(storage.GetItem(0));
//  `int` should be stored in an allocated box
TEST_ExpectNotNone(box);
TEST_ExpectTrue(box.IsAllocated());
//  Re-recording `int` as ref causes previous box (managed by `storage`)
//  to get destroyed
storage.SetInt(0, 11, true);
TEST_ExpectNotNone(box);                // still not `none`
TEST_ExpectFalse(box.IsAllocated());    // but is not deallocated
Log("Value as int:" @ storage.GetInt(0)); //  Value as int: 11
//  `int` should be stored in an allocated ref now
ref = IntRef(storage.GetItem(0));
TEST_ExpectNotNone(ref);
TEST_ExpectTrue(ref.IsAllocated());

Even more accessors

Collections DynamicArray and AssociativeArray are AcediaObjects themselves and, therefore, can be stored in other arrays, producing hierarchical structures, similar to those of JSON's arrays / objects.

{
    "main_guy": {
        "status": "admin",
        "maps": ["biotics", "bedlam", "waterworks"]
    },
    "other_guy": {
        "status": "random",
        "maps": ["biotics", "westlondon"]
    }
}

To access some variable, nested deep inside such structure, one can either manually get reference of each collection on the way, e.g. to access second map of the "other_guy" we'd need to first get reference to "other_guy"'s collection (AssociativeArray):

{
    "status": "random",
    "maps": ["biotics", "westlondon"]
}

then to the array of his maps (DynamicArray):

["biotics", "westlondon"]

and only then access second item. This is too cumbersome! Fortunately, Acedia's collections have an alternative solution:

userCollection.GetTextBy(P("/other_guy/maps/1"));   //  westlondon!

/other_guy/maps/1 line is describes a path to the element nested deep inside hierarchy of collections and follows the rules of a JSON pointer. Both DynamicArray and AssociativeArray support following methods that work with such pointers: GetItemBy(), GetBoolBy(), GetByteBy(), GetIntBy(), GetFloatBy(), GetTextBy(), GetDynamicArrayBy() and GetAssociativeArrayBy().

Passing paths like /other_guy/maps/1 requires collections to perform their parsing every time such getter is called. If you want to reuse the same path several times it might be better to convert it into JSONPointer object (using _.json.Pointer() method) and then use that object with following alternative methods: GetItemByJSON(), GetBoolByJSON(), GetByteByJSON(), GetIntByJSON(), GetFloatByJSON(), GetTextByJSON(), GetDynamicArrayByJSON(), GetAssociativeArrayByJSON(). This way parsing has to be done only once - when creating JSONPointer object.