Abstract superclass of all array node types (flat hierarchy). Any Content can be nested within any other Content. More...
#include <Content.h>
Public Member Functions | |
Content (const IdentitiesPtr &identities, const util::Parameters ¶meters) | |
Called by all subclass constructors; assigns identities and parameters upon construction. More... | |
virtual | ~Content () |
Empty destructor; required for some C++ reason. More... | |
virtual bool | isscalar () const |
Returns true if the data represented by this node is scalar, not a true array. More... | |
virtual const std::string | classname () const =0 |
User-friendly name of this class, including integer-type specialization. More... | |
virtual const IdentitiesPtr | identities () const |
Optional Identities for each element of the array (may be nullptr ). More... | |
virtual void | setidentities ()=0 |
Assign a surrogate index of Identities to this array (in-place). More... | |
virtual void | setidentities (const IdentitiesPtr &identities)=0 |
Assign a specified set of Identities to this array (in-place). More... | |
virtual const TypePtr | type (const util::TypeStrs &typestrs) const =0 |
High-level Type describing this array. More... | |
virtual const FormPtr | form (bool materialize) const =0 |
Low-level Form describing all the features of this array except the actual data buffers (Index , Identities , NumpyArray::ptr , and RawArray::ptr ). More... | |
virtual kernel::lib | kernels () const =0 |
Returns the kernel library enum for all nested ptr_lib within the array's tree structure. If different libraries are associated with nodes of the same tree, this returns kernel::lib::size to indicate that it is "mixed". More... | |
virtual void | caches (std::vector< ArrayCachePtr > &out) const =0 |
Accumulates all the unique #ArrayCache objects from nested #VirtualArray nodes. (Uniqueness is determined by pointer value.) More... | |
virtual const std::string | tostring_part (const std::string &indent, const std::string &pre, const std::string &post) const =0 |
Internal function to build an output string for tostring. More... | |
virtual void | tojson_part (ToJson &builder, bool include_beginendlist) const =0 |
Internal function to produce a JSON representation one node at a time. More... | |
virtual void | nbytes_part (std::map< size_t, int64_t > &largest) const =0 |
Internal function used to calculate nbytes. More... | |
virtual int64_t | length () const =0 |
The number of elements in the array. More... | |
virtual const ContentPtr | shallow_copy () const =0 |
Copies this node without copying any nodes hierarchically nested within it or any array/index/identity buffers. More... | |
virtual const ContentPtr | deep_copy (bool copyarrays, bool copyindexes, bool copyidentities) const =0 |
Copies this node and all nodes hierarchically nested within it, optionally copying the associated arrays, indexes, and identities, too. More... | |
virtual void | check_for_iteration () const =0 |
Performs up-front validity checks on an array so that they don't have to be checked in getitem_at_nowrap for each item. More... | |
virtual const ContentPtr | getitem_nothing () const =0 |
Internal function to get an empty slice (with the correct type). More... | |
virtual const ContentPtr | getitem_at (int64_t at) const =0 |
Returns the element at a given position in the array, handling negative indexing and bounds-checking like Python. More... | |
virtual const ContentPtr | getitem_at_nowrap (int64_t at) const =0 |
Returns the element at a given position in the array, without handling negative indexing or bounds-checking. More... | |
virtual const ContentPtr | getitem_range (int64_t start, int64_t stop) const =0 |
Subinterval of this array, handling negative indexing and bounds-checking like Python. More... | |
virtual const ContentPtr | getitem_range_nowrap (int64_t start, int64_t stop) const =0 |
Subinterval of this array, without handling negative indexing or bounds-checking. More... | |
virtual const ContentPtr | getitem_field (const std::string &key) const =0 |
This array with the first nested RecordArray replaced by the field at key . More... | |
virtual const ContentPtr | getitem_field (const std::string &key, const Slice &only_fields) const =0 |
virtual const ContentPtr | getitem_fields (const std::vector< std::string > &keys) const =0 |
This array with the first nested RecordArray replaced by a RecordArray of a given subset of keys . More... | |
virtual const ContentPtr | getitem_fields (const std::vector< std::string > &keys, const Slice &only_fields) const =0 |
virtual const ContentPtr | getitem (const Slice &where) const |
Entry point for general slicing: Slice represents a tuple of SliceItem nodes applying to each level of nested lists. More... | |
virtual const ContentPtr | getitem_next (const SliceItemPtr &head, const Slice &tail, const Index64 &advanced) const |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceItemPtr &slicecontent, const Slice &tail) const |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More... | |
virtual const ContentPtr | carry (const Index64 &carry, bool allow_lazy) const =0 |
Returns an array of the same type with elements filtered, rearranged, and possibly duplicated by the carry array of integers. More... | |
virtual const std::string | purelist_parameter (const std::string &key) const |
The parameter associated with key at the first level that has a non-null value, descending only as deep as the first RecordArray. More... | |
bool | purelist_isregular () const |
Returns true if all nested lists down to the first RecordArray are RegularArray nodes; false otherwise. More... | |
virtual bool | dimension_optiontype () const |
Returns true if this dimension has option-type; false otherwise. More... | |
virtual int64_t | purelist_depth () const =0 |
The list-depth of this array, not counting any contained within a RecordArray. More... | |
virtual const std::pair< int64_t, int64_t > | minmax_depth () const =0 |
Returns (a) the minimum list-depth and (b) the maximum list-depth of the array, which can differ if this array "branches" (differs when followed through different fields of a RecordArray or UnionArray ). More... | |
virtual const std::pair< bool, int64_t > | branch_depth () const =0 |
Returns (a) whether the list-depth of this array "branches," or differs when followed through different fields of a RecordArray or UnionArray and (b) the minimum list-depth. More... | |
virtual int64_t | numfields () const =0 |
The number of fields in the first nested tuple or records or -1 if this array does not contain a RecordArray. More... | |
virtual int64_t | fieldindex (const std::string &key) const =0 |
The position of a tuple or record key name if this array contains a RecordArray. More... | |
virtual const std::string | key (int64_t fieldindex) const =0 |
The record name associated with a given field index or the tuple index as a string (e.g. "0" , "1" , "2" ) if a tuple. More... | |
virtual bool | haskey (const std::string &key) const =0 |
Returns true if the array contains a RecordArray with the specified key ; false otherwise. More... | |
virtual const std::vector< std::string > | keys () const =0 |
A list of RecordArray keys or an empty list if this array does not contain a RecordArray. More... | |
virtual bool | istuple () const =0 |
Returns true if the outermost RecordArray is a tuple. More... | |
virtual const std::string | validityerror (const std::string &path) const =0 |
Returns an error message if this array is invalid; otherwise, returns an empty string. More... | |
virtual const ContentPtr | shallow_simplify () const =0 |
Returns an equivalent array simplified at one level only using simplify_optiontype if an option-type array and simplify_uniontype if a union-type array. More... | |
virtual const ContentPtr | num (int64_t axis, int64_t depth) const =0 |
The length of this array (as a NumpyArray scalar) if axis = 0 or the lengths of subarrays (as an array or nested array) if axis != 0 . More... | |
virtual const std::pair< Index64, ContentPtr > | offsets_and_flattened (int64_t axis, int64_t depth) const =0 |
Returns (a) an offsets Index and (b) a flattened version of the array at some axis depth. More... | |
virtual bool | mergeable (const ContentPtr &other, bool mergebool) const =0 |
Returns true if this array can be merged with the other ; false otherwise. More... | |
virtual bool | referentially_equal (const ContentPtr &other) const =0 |
Returns true if this array has all the same buffers and parameters as other ; false otherwise. More... | |
virtual const std::pair< ContentPtrVec, ContentPtrVec > | merging_strategy (const ContentPtrVec &others) const |
Partitions this array plus a list of others into a head sequence and a tail sequence: More... | |
virtual const ContentPtr | reverse_merge (const ContentPtr &other) const |
Merges a single other with this array in reverse order: other first, this last. More... | |
const ContentPtr | merge (const ContentPtr &other) const |
An array with this and the other concatenated (this first, other last). More... | |
virtual const ContentPtr | mergemany (const ContentPtrVec &others) const =0 |
Returns an array with this and the others concatenated (in order, this first, others last). More... | |
virtual const SliceItemPtr | asslice () const =0 |
Converts this array into a SliceItem that can be used in getitem. More... | |
virtual const ContentPtr | fillna (const ContentPtr &value) const =0 |
Returns this array with None values replaced by a given value . More... | |
virtual const ContentPtr | rpad (int64_t target, int64_t axis, int64_t depth) const =0 |
If axis = 0 , returns a view of this array padded on the right with None values to have a minimum length; otherwise, returns an array with nested lists all padded to the minimum length. More... | |
virtual const ContentPtr | rpad_and_clip (int64_t target, int64_t axis, int64_t depth) const =0 |
If axis = 0 , returns a view of this array padded on the right. More... | |
virtual const ContentPtr | reduce_next (const Reducer &reducer, int64_t negaxis, const Index64 &starts, const Index64 &shifts, const Index64 &parents, int64_t outlength, bool mask, bool keepdims) const =0 |
This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all). More... | |
virtual const ContentPtr | sort_next (int64_t negaxis, const Index64 &starts, const Index64 &parents, int64_t outlength, bool ascending, bool stable) const =0 |
This array sorted. More... | |
virtual const ContentPtr | argsort_next (int64_t negaxis, const Index64 &starts, const Index64 &shifts, const Index64 &parents, int64_t outlength, bool ascending, bool stable) const =0 |
This array sorted indices. More... | |
virtual const ContentPtr | localindex (int64_t axis, int64_t depth) const =0 |
A (possibly nested) array of integers indicating the positions of elements within each nested list. More... | |
virtual const ContentPtr | combinations (int64_t n, bool replacement, const util::RecordLookupPtr &recordlookup, const util::Parameters ¶meters, int64_t axis, int64_t depth) const =0 |
Tuples or records of all n -tuple combinations of list items at some axis depth. More... | |
const std::string | tostring () const |
Returns a string representation of this array (multi-line XML). More... | |
const std::string | tojson (bool pretty, int64_t maxdecimals, const char *nan_string=nullptr, const char *infinity_string=nullptr, const char *minus_infinity_string=nullptr, const char *complex_real_string=nullptr, const char *complex_imag_string=nullptr) const |
Returns a JSON representation of this array. More... | |
void | tojson (FILE *destination, bool pretty, int64_t maxdecimals, int64_t buffersize, const char *nan_string=nullptr, const char *infinity_string=nullptr, const char *minus_infinity_string=nullptr, const char *complex_real_string=nullptr, const char *complex_imag_string=nullptr) const |
Writes a JSON representation of this array to a destination file. More... | |
int64_t | nbytes () const |
The number of bytes contained in all array buffers, Index buffers, and Identities buffers, not including the lightweight node objects themselves. More... | |
const ContentPtr | reduce (const Reducer &reducer, int64_t axis, bool mask, bool keepdims) const |
This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all). More... | |
const ContentPtr | sort (int64_t axis, bool ascending, bool stable) const |
This array with one axis sorted by applying a sorting algorithm. More... | |
const ContentPtr | argsort (int64_t axis, bool ascending, bool stable) const |
This array indices with one axis sorted by applying a sorting algorithm. More... | |
const util::Parameters | parameters () const |
String-to-JSON map that augments the meaning of this array. More... | |
void | setparameters (const util::Parameters ¶meters) |
Assign all the parameters for this array node (in-place). More... | |
const std::string | parameter (const std::string &key) const |
Get one parameter from this array node. More... | |
void | setparameter (const std::string &key, const std::string &value) |
Assign one parameter for this array node (in-place). More... | |
bool | parameter_equals (const std::string &key, const std::string &value) const |
Returns true if the parameter associated with key exists and is equal to value ; false otherwise. More... | |
bool | parameters_equal (const util::Parameters &other, bool check_all) const |
Returns true if all parameters of this array node are equal to the other parameters. More... | |
bool | parameter_isstring (const std::string &key) const |
Returns true if the parameter associated with key is a string; false otherwise. More... | |
bool | parameter_isname (const std::string &key) const |
Returns true if the parameter associated with key is a string that matches [A-Za-z_][A-Za-z_0-9]* ; false otherwise. More... | |
const std::string | parameter_asstring (const std::string &key) const |
Returns the parameter associated with key as a string if parameter_isstring; raises an error otherwise. More... | |
const ContentPtr | merge_as_union (const ContentPtr &other) const |
Concatenates this array with other by creating a UnionArray instead of actually merging the data. This does not call UnionArray::simplify_uniontype . More... | |
const ContentPtr | rpad_axis0 (int64_t target, bool clip) const |
Internal function to handle the axis = 0 case of rpad and rpad_and_clip. More... | |
const ContentPtr | localindex_axis0 () const |
Internal function to handle the axis = 0 case of localindex. More... | |
const ContentPtr | combinations_axis0 (int64_t n, bool replacement, const util::RecordLookupPtr &recordlookup, const util::Parameters ¶meters) const |
Internal function to handle the axis = 0 case of combinations. More... | |
virtual const ContentPtr | getitem_next (const SliceAt &at, const Slice &tail, const Index64 &advanced) const =0 |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceRange &range, const Slice &tail, const Index64 &advanced) const =0 |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceEllipsis &ellipsis, const Slice &tail, const Index64 &advanced) const |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceNewAxis &newaxis, const Slice &tail, const Index64 &advanced) const |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceArray64 &array, const Slice &tail, const Index64 &advanced) const =0 |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceField &field, const Slice &tail, const Index64 &advanced) const |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceFields &fields, const Slice &tail, const Index64 &advanced) const |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceMissing64 &missing, const Slice &tail, const Index64 &advanced) const |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next (const SliceJagged64 &jagged, const Slice &tail, const Index64 &advanced) const =0 |
Internal function that propagates a generic getitem request through one axis (including advanced indexing). More... | |
virtual const ContentPtr | getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceArray64 &slicecontent, const Slice &tail) const =0 |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More... | |
virtual const ContentPtr | getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceMissing64 &slicecontent, const Slice &tail) const =0 |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More... | |
virtual const ContentPtr | getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceJagged64 &slicecontent, const Slice &tail) const =0 |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More... | |
const int64_t | axis_wrap_if_negative (int64_t axis) const |
Internal function defining the negative axis handling for many operations. More... | |
virtual const ContentPtr | copy_to (kernel::lib ptr_lib) const =0 |
Recursively copies components of the array from main memory to a GPU (if ptr_lib == kernel::lib::cuda ) or to main memory (if ptr_lib == kernel::lib::cpu ) if those components are not already there. More... | |
virtual const ContentPtr | numbers_to_type (const std::string &name) const =0 |
Change the leaf types to 'totype'. More... | |
virtual bool | is_unique () const =0 |
Returns 'true' if all components of the array are unique. More... | |
virtual const ContentPtr | unique () const =0 |
Returns sorted unique values. More... | |
virtual bool | is_subrange_equal (const Index64 &start, const Index64 &stop) const =0 |
Returns 'true' if subranges are equal. More... | |
Protected Member Functions | |
const std::string | validityerror_parameters (const std::string &path) const |
Internal function to determine if a particular set of parameters are allowed on a particular node (centralized by parameter, not by class). More... | |
const ContentPtr | getitem_next_array_wrap (const ContentPtr &outcontent, const std::vector< int64_t > &shape) const |
Internal function to wrap putative getitem output with enough RegularArray nodes to satisfy a given shape . More... | |
const std::string | parameters_tostring (const std::string &indent, const std::string &pre, const std::string &post) const |
Internal function to convert parameters into a string fragment for tostring. More... | |
kernel::lib | kernels_compare (kernel::lib from_index, const ContentPtr &content) const |
Internal function to support most of the logic in kernels. More... | |
Protected Attributes | |
IdentitiesPtr | identities_ |
See identities. More... | |
util::Parameters | parameters_ |
See parameters. More... | |
Abstract superclass of all array node types (flat hierarchy). Any Content can be nested within any other Content.
Content | ( | const IdentitiesPtr & | identities, |
const util::Parameters & | parameters | ||
) |
Called by all subclass constructors; assigns identities and parameters upon construction.
|
inlinevirtual |
Empty destructor; required for some C++ reason.
const ContentPtr argsort | ( | int64_t | axis, |
bool | ascending, | ||
bool | stable | ||
) | const |
This array indices with one axis sorted by applying a sorting algorithm.
This operation is implemented on each node through argsort_next.
axis | The axis to sort. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
ascending | If true , the values will be sorted in an ascending order. |
stable | If true , the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values). |
|
pure virtual |
This array sorted indices.
The user's entry point for this operation is argsort.
negaxis | The negative axis: -axis . That is, negaxis = 1 means the deepest axis level. |
starts | Staring positions of each group to combine as an Index . These are downward pointers from an outer structure into this structure with the same meaning as in ListArray . |
shifts | Per-element adjustments that allows for variable-length lists with axis != -1 and for missing values (None). |
parents | Groups to combine as an Index of upward pointers from this structure to the outer structure to sort. |
outlength | The length of the array, after the operation completes. |
ascending | If true , the values will be sorted in an ascending order. |
stable | If true , the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Converts this array into a SliceItem that can be used in getitem.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const int64_t axis_wrap_if_negative | ( | int64_t | axis | ) | const |
Internal function defining the negative axis handling for many operations.
Returns a non-negative equivalent axis
if unambiguous and passes the negative axis
through if ambiguous. A negative axis
can be ambiguous if the list-depth "branches" in a RecordArray or a UnionArray
with different contents
having different depths. As an operation descends through the nodes of an array, it repeatedly calls axis_wrap_if_negative
until an unambiguous non-negative axis
can be identified.
This allows a RecordArray or UnionArray
with different depths to accept axis = -1
as the last axis, regardless of how deep that is in different record fields or union possibilities.
|
pure virtual |
Returns (a) whether the list-depth of this array "branches," or differs when followed through different fields of a RecordArray or UnionArray
and (b) the minimum list-depth.
If the array does not contain any records or heterogeneous data, the first
element is always true
and the second
is simply the depth.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Accumulates all the unique #ArrayCache objects from nested #VirtualArray nodes. (Uniqueness is determined by pointer value.)
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns an array of the same type with elements filtered, rearranged, and possibly duplicated by the carry
array of integers.
The output has the same length as the carry
index, not the array
that is being manipulated. For each item i
in carry
, the output is array[index[i]]
.
This operation is called take in NumPy and Arrow, although this carry is a low-level function that does not handle negative indexes and is not exposed to the Python layer. It is used by many operations to pass filters/rearrangements/duplications from one typed array node to another without knowing the latter's type.
Taking getitem_at_nowrap as a function from integers to the array's item type, A: [0, len(a)) → T
, and the carry
array's Index64::getitem_at_nowrap
as a function C: [0, len(c)) → [0, len(a))
, this method represents function composition, A ∘ C: [0, len(c)) → T
.
carry
array has duplicate elements, the array is only duplicated one level deep. For instance, on a ListArray
, only the starts
and stops
get duplicated, not the content
(and similarly for all other node types). Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Performs up-front validity checks on an array so that they don't have to be checked in getitem_at_nowrap for each item.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
User-friendly name of this class, including integer-type specialization.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Tuples or records of all n
-tuple combinations of list items at some axis
depth.
For example, the n = 2
combinations at axis = 0
of
would be
and the n = 3
combinations at axis = 1
of
would be
n | The number of items in each tuple/record. |
replacement | If true , the tuples/records are allowed to include the same item more than once, such as (a, a, a) and (a, a, b) . In the above examples, replacement = false . (If the output of this function is thought of as the "upper triangle" elements of the Cartesian product of the input with itself n times, the replacement parameter determines whether elements on the diagonal are allowed.) |
recordlookup | If nullptr , the output consists of tuples, a RecordArray indexed by "0" , "1" , "2" , etc. If recordlookup is a std::vector<std::string> , the output consists of records, a RecordArray indexed by names (strings). The length of recordlookup must be equal to n . |
parameters | Parameters assigned to the new RecordArray. This can be used to set "__record_" = "\"record_name\\"" to give the records a custom behavior in Python. |
axis | The nesting depth at which this operation is applied. At axis = 0 , combinations are formed from elements of the whole array (see example above); at axis = 1 , combinations are formed independently for each element. At a deeper axis , outer levels are left untouched. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
depth | The current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis ). |
Note that axis = 0
is qualitatively different from any other axis
because a dataset is typically much larger than any one of its elements. As such, axis = 0
is lazily generated with an IndexedArray
, while any other axis
is eagerly generated by duplicating elements (with carry).
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const ContentPtr combinations_axis0 | ( | int64_t | n, |
bool | replacement, | ||
const util::RecordLookupPtr & | recordlookup, | ||
const util::Parameters & | parameters | ||
) | const |
Internal function to handle the axis = 0
case of combinations.
The axis = 0
case does not depend on array node type, so it is defined universally in the Content class.
n | The number of items in each tuple/record. |
replacement | If true , the tuples/records are allowed to include the same item more than once, such as (a, a, a) and (a, a, b) . In the above examples, replacement = false . (If the output of this function is thought of as the "upper triangle" elements of the Cartesian product of the input with itself n times, the replacement parameter determines whether elements on the diagonal are allowed.) |
recordlookup | If nullptr , the output consists of tuples, a RecordArray indexed by "0" , "1" , "2" , etc. If recordlookup is a std::vector<std::string> , the output consists of records, a RecordArray indexed by names (strings). The length of recordlookup must be equal to n . |
parameters | Parameters assigned to the new RecordArray. This can be used to set "__record_" = "\"record_name\\"" to give the records a custom behavior in Python. |
|
pure virtual |
Recursively copies components of the array from main memory to a GPU (if ptr_lib == kernel::lib::cuda
) or to main memory (if ptr_lib == kernel::lib::cpu
) if those components are not already there.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Copies this node and all nodes hierarchically nested within it, optionally copying the associated arrays, indexes, and identities, too.
See also shallow_copy.
copyarrays | If true , copy the associated array buffers (in NumpyArray and RawArray ), not just the lightweight objects that point to them. |
copyindexes | If true , copy the Index objects and their buffers as well. |
copyidentities | If true , copy the Identities objects and their buffers as well. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Returns true
if this dimension has option-type; false
otherwise.
Reimplemented in Record.
|
pure virtual |
The position of a tuple or record key name if this array contains a RecordArray.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns this array with None
values replaced by a given value
.
value | An array of exactly one element, which need not have the same type as the missing values it's replacing. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Low-level Form describing all the features of this array except the actual data buffers (Index
, Identities
, NumpyArray::ptr
, and RawArray::ptr
).
materialize | If true, materialize any VirtualArrays encountered along the way. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Entry point for general slicing: Slice represents a tuple of SliceItem nodes applying to each level of nested lists.
Reimplemented in NumpyArray, Record, and VirtualArray.
|
pure virtual |
Returns the element at a given position in the array, handling negative indexing and bounds-checking like Python.
The first item in the array is at 0
, the second at 1
, the last at -1
, the penultimate at -2
, etc.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns the element at a given position in the array, without handling negative indexing or bounds-checking.
If the array has Identities, the identity bounds are checked.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
This array with the first nested RecordArray replaced by the field at key
.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
|
pure virtual |
This array with the first nested RecordArray replaced by a RecordArray of a given subset of keys
.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
|
pure virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
|
virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
Reimplemented in EmptyArray, None, NumpyArray, Record, RecordArray, and VirtualArray.
|
virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
Reimplemented in EmptyArray, None, NumpyArray, Record, RecordArray, and VirtualArray.
|
virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
head | First element of the Slice tuple. |
tail | The rest of the elements of the Slice tuple. |
advanced | If empty, no array slices (integer or boolean) have been encountered yet; otherwise, positions in any subsequent array slices to select. |
In the NumPy documentation, advanced indexes are described as iterating "as one," which requires an Index
to be propagated when implemented recursively.
Reimplemented in BitMaskedArray, ByteMaskedArray, IndexedArrayOf< T, ISOPTION >, NumpyArray, RecordArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
|
virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
|
pure virtual |
Internal function that propagates a generic getitem request through one axis (including advanced indexing).
See generic getitem_next for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
protected |
Internal function to wrap putative getitem output with enough RegularArray nodes to satisfy a given shape
.
The shape
is intended to match a SliceArray::shape
.
|
pure virtual |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.
See generic getitem_next_jagged for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.
slicestarts | Effective starts (similar to ListArray 's starts ) of the jagged slice. |
slicestops | Effective stops of the jagged slice. |
slicecontent | Nested content within the jagged slice. |
tail | Subsequent SliceItem elements beyond the jagged array hierarchy. |
Reimplemented in ListOffsetArrayOf< T >, and VirtualArray.
|
pure virtual |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.
See generic getitem_next_jagged for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.
See generic getitem_next_jagged for details.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Internal function to get an empty slice (with the correct type).
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Subinterval of this array, handling negative indexing and bounds-checking like Python.
The first item in the array is at 0
, the second at 1
, the last at -1
, the penultimate at -2
, etc.
Ranges beyond the array are not an error; they are trimmed to start = 0
on the left and stop = length() - 1
on the right.
This operation only affects the node metadata; its calculation time does not scale with the size of the array.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Subinterval of this array, without handling negative indexing or bounds-checking.
If the array has Identities, the identity bounds are checked.
This operation only affects the node metadata; its calculation time does not scale with the size of the array.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns true
if the array contains a RecordArray with the specified key
; false
otherwise.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Optional Identities for each element of the array (may be nullptr
).
Reimplemented in Record.
Returns 'true' if subranges are equal.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, VirtualArray, and NumpyArray.
|
pure virtual |
Returns 'true' if all components of the array are unique.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Returns true
if the data represented by this node is scalar, not a true array.
Scalars are not valid inputs to many operations, but they are instances of Content so that they can be outputs of some operations.
Formally, scalars and arrays should probably be distinct C++ types to eliminate this ambiguity, but that would introduce excessive type-testing and casting in all the algorithms that chain operations (output of one is input to another). It is more convenient to test for scalars at runtime with this method.
The only scalars are:
shape.empty()
(zero-dimensional array), which represents scalar numbers and booleans. (Immediately converted to a number object in Python.)None
in Python.)Reimplemented in None, NumpyArray, and Record.
|
pure virtual |
Returns true
if the outermost RecordArray is a tuple.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns the kernel library enum for all nested ptr_lib
within the array's tree structure. If different libraries are associated with nodes of the same tree, this returns kernel::lib::size
to indicate that it is "mixed".
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
protected |
Internal function to support most of the logic in kernels.
|
pure virtual |
The record name associated with a given field index or the tuple index as a string (e.g. "0"
, "1"
, "2"
) if a tuple.
Raises an error if the array does not contain a RecordArray.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
A list of RecordArray keys or an empty list if this array does not contain a RecordArray.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
The number of elements in the array.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
A (possibly nested) array of integers indicating the positions of elements within each nested list.
axis | The nesting depth at which this operation is applied. If axis = 0 , the output is simply an array of integers from 0 to length() . If axis = 1 , the output has one level of nesting containing integers from 0 to the length of the nested list. Higher values of axis leave outer layers of the structure untouched. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
depth | The current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis ). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const ContentPtr localindex_axis0 | ( | ) | const |
Internal function to handle the axis = 0
case of localindex.
The axis = 0
case does not depend on array node type, so it is defined universally in the Content class.
const ContentPtr merge | ( | const ContentPtr & | other | ) | const |
An array with this and the other
concatenated (this first, other
last).
const ContentPtr merge_as_union | ( | const ContentPtr & | other | ) | const |
Concatenates this array with other
by creating a UnionArray
instead of actually merging the data. This does not call UnionArray::simplify_uniontype
.
|
pure virtual |
Returns true
if this array can be merged with the other
; false
otherwise.
The merge method will complete without errors if this function returns true
.
other | The other array to merge with. |
mergebool | If true , consider boolean types to be equivalent to integers. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns an array with this and the others
concatenated (in order, this first, others
last).
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Partitions this
array plus a list of others
into a head
sequence and a tail
sequence:
[this] + others == head + tail
The order is preserved and no arrays are lost. The others
must have at least one array, and this
will be added to head
, though tail
might be empty.
(If tail
is empty, head
must contain at least two arrays!)
The position of the split between head
and tail
is such that the first array in tail
must be merged with reverse_merge (usually because it is an option-type or union-type array, though this rule depends on whether this
is option-type, union-type, or neither).
The purpose of this function is to prepare a head
of arrays that can all be merged "normally," followed by a tail
whose first array must be "reverse merged" and subsequent arrays recurse.
This is the first step in merge.
Reimplemented in IndexedArrayOf< T, ISOPTION >, and UnionArrayOf< T, I >.
|
pure virtual |
Returns (a) the minimum list-depth and (b) the maximum list-depth of the array, which can differ if this array "branches" (differs when followed through different fields of a RecordArray or UnionArray
).
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
int64_t nbytes | ( | ) | const |
The number of bytes contained in all array buffers, Index
buffers, and Identities buffers, not including the lightweight node objects themselves.
|
pure virtual |
Internal function used to calculate nbytes.
largest | The largest range of bytes used in each reference-counted pointer (size_t ). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
The length of this array (as a NumpyArray scalar) if axis = 0
or the lengths of subarrays (as an array or nested array) if axis != 0
.
axis | The axis whose length or lengths to quantify. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
depth | The current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis ). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Change the leaf types to 'totype'.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
The number of fields in the first nested tuple or records or -1
if this array does not contain a RecordArray.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns (a) an offsets Index
and (b) a flattened version of the array at some axis
depth.
If axis > 1
(or its negative equivalent), the offsets is empty.
axis | The axis to eliminate by flattening. axis = 0 is invalid. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
depth | The current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis ). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const std::string parameter | ( | const std::string & | key | ) | const |
Get one parameter from this array node.
If the key
does not exist, this function returns "null"
.
Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value\""
.
const std::string parameter_asstring | ( | const std::string & | key | ) | const |
Returns the parameter associated with key
as a string if parameter_isstring; raises an error otherwise.
bool parameter_equals | ( | const std::string & | key, |
const std::string & | value | ||
) | const |
Returns true
if the parameter associated with key
exists and is equal to value
; false
otherwise.
Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value\""
.
Equality is checked at the level of JSON DOMs. The value
does not need to be exactly the same string; it needs to have equivalent JSON value.
bool parameter_isname | ( | const std::string & | key | ) | const |
Returns true
if the parameter associated with key
is a string that matches [A-Za-z_][A-Za-z_0-9]*
; false
otherwise.
bool parameter_isstring | ( | const std::string & | key | ) | const |
Returns true
if the parameter associated with key
is a string; false
otherwise.
const util::Parameters parameters | ( | ) | const |
String-to-JSON map that augments the meaning of this array.
Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value\""
.
bool parameters_equal | ( | const util::Parameters & | other, |
bool | check_all | ||
) | const |
Returns true
if all parameters of this array node are equal to the other
parameters.
Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value\""
.
Equality is checked at the level of JSON DOMs. The value
does not need to be exactly the same string; it needs to have equivalent JSON value.
If check_all
, every parameter is checked; otherwise, only "__array__"
and "__record__"
are checked.
|
protected |
Internal function to convert parameters into a string fragment for tostring.
indent | Indentation depth as a string of spaces. |
pre | Prefix string, usually an opening XML tag. |
post | Postfix string, usually a closing XML tag and carriage return. |
|
pure virtual |
The list-depth of this array, not counting any contained within a RecordArray.
The purelist_depth
of a Record is 0
, and a RecordArray is 1
(regardless of what its fields contain).
If this array contains a UnionArray
with different depths, the return value is -1
.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
bool purelist_isregular | ( | ) | const |
Returns true
if all nested lists down to the first RecordArray are RegularArray nodes; false
otherwise.
|
virtual |
The parameter associated with key
at the first level that has a non-null value, descending only as deep as the first RecordArray.
const ContentPtr reduce | ( | const Reducer & | reducer, |
int64_t | axis, | ||
bool | mask, | ||
bool | keepdims | ||
) | const |
This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all).
This operation is implemented on each node through reduce_next.
reducer | The choice of Reducer algorithm. |
axis | The axis to remove by reduction. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
mask | If true , the Reducer's identity values will be covered by None using a ByteMaskedArray. This is desirable for ReducerMin, ReducerMax, ReducerArgmin, and ReducerArgmax to indicate that empty lists have no minimum or maximum. |
keepdims | If true , the reduced values will be wrapped by a singleton RegularArray to maintain the same number of dimensions in the output. |
|
pure virtual |
This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all).
The user's entry point for this operation is reduce.
reducer | The choice of Reducer algorithm. |
negaxis | The negative axis: -axis . That is, negaxis = 1 means the deepest axis level. |
starts | Staring positions of each group to combine as an Index . These are downward pointers from an outer structure into this structure with the same meaning as in ListArray . Only used by a reducer that returns_positions (currently only argmin and argmax). |
shifts | Per-element adjustments for any reducer that returns_positions . Allows for variable-length lists with axis != -1 and for missing values (None) in argmin and argmax. |
parents | Groups to combine as an Index of upward pointers from this structure to the outer structure to reduce. |
outlength | The length of the reduced array, after the operation completes. |
mask | If true , the Reducer's identity values will be covered by None using a ByteMaskedArray. This is desirable for ReducerMin, ReducerMax, ReducerArgmin, and ReducerArgmax to indicate that empty lists have no minimum or maximum. |
keepdims | If true , the reduced values will be wrapped by a singleton RegularArray to maintain the same number of dimensions in the output. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns true
if this array has all the same buffers and parameters as other
; false
otherwise.
other | The array to compare this with. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
virtual |
Merges a single other
with this array in reverse order: other
first, this last.
Only arrays that need to be reversible have this function: option-type and union-type arrays. Others raise a runtime error.
Reimplemented in BitMaskedArray, ByteMaskedArray, IndexedArrayOf< T, ISOPTION >, UnionArrayOf< T, I >, and UnmaskedArray.
|
pure virtual |
If axis = 0
, returns a view of this array padded on the right with None
values to have a minimum length; otherwise, returns an array with nested lists all padded to the minimum length.
target | The intended length. The output may be longer than this target length, but not shorter (using ListArray ). |
axis | The axis at which to apply padding. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
depth | The current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis ). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
If axis = 0
, returns a view of this array padded on the right.
with None
values to have exactly the specified length; otherwise, returns an array with nested lists all padded to the specified length.
target | The intended length; the output has exactly this target length (using RegularArray). |
axis | The axis at which to apply padding. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
depth | The current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis ). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const ContentPtr rpad_axis0 | ( | int64_t | target, |
bool | clip | ||
) | const |
Internal function to handle the axis = 0
case of rpad and rpad_and_clip.
The axis = 0
case does not depend on array node type, so it is defined universally in the Content class.
target | The intended length. The output may be longer than this target length, but not shorter (using ListArray ). |
clip | If true , do rpad_and_clip; if false , do rpad. |
|
pure virtual |
Assign a surrogate index of Identities to this array (in-place).
This also assigns and possibly replaces Identities in nested arrays.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Assign a specified set of Identities to this array (in-place).
This also assigns and possibly replaces Identities in nested arrays.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
void setparameter | ( | const std::string & | key, |
const std::string & | value | ||
) |
Assign one parameter for this array node (in-place).
Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value\""
.
void setparameters | ( | const util::Parameters & | parameters | ) |
Assign all the parameters for this array node (in-place).
Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value\""
.
|
pure virtual |
Copies this node without copying any nodes hierarchically nested within it or any array/index/identity buffers.
See also deep_copy.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns an equivalent array simplified at one level only using simplify_optiontype
if an option-type array and simplify_uniontype
if a union-type array.
For all other types of arrays, this operation is a pass-through.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const ContentPtr sort | ( | int64_t | axis, |
bool | ascending, | ||
bool | stable | ||
) | const |
This array with one axis sorted by applying a sorting algorithm.
This operation is implemented on each node through sort_next.
axis | The axis to sort. Negative axis counts backward from the deepest levels (-1 is the last valid axis ). |
ascending | If true , the values will be sorted in an ascending order. |
stable | If true , the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values). |
|
pure virtual |
This array sorted.
The user's entry point for this operation is sort.
negaxis | The negative axis: -axis . That is, negaxis = 1 means the deepest axis level. |
starts | Staring positions of each group to combine as an Index . These are downward pointers from an outer structure into this structure with the same meaning as in ListArray . |
parents | Groups to combine as an Index of upward pointers from this structure to the outer structure to sort. |
outlength | The length of the array, after the operation completes. |
ascending | If true , the values will be sorted in an ascending order. |
stable | If true , the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values). |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const std::string tojson | ( | bool | pretty, |
int64_t | maxdecimals, | ||
const char * | nan_string = nullptr , |
||
const char * | infinity_string = nullptr , |
||
const char * | minus_infinity_string = nullptr , |
||
const char * | complex_real_string = nullptr , |
||
const char * | complex_imag_string = nullptr |
||
) | const |
Returns a JSON representation of this array.
pretty | If true , add spacing to make the JSON human-readable. If false , return a compact representation. |
maxdecimals | Maximum number of decimals for floating-point numbers or -1 for no limit. |
nan_string | user-defined string for a not-a-number (NaN) value representation in JSON format |
infinity_string | user-defined string for a positive infinity representation in JSON format |
minus_infinity_string | user-defined string for a negative infinity representation in JSON format |
Although the JSON output contains all of the data from the array (and therefore might not be safe to print to the screen), it does not preserve the type information of an array. In particular, the distinction between ListType and RegularType is lost.
void tojson | ( | FILE * | destination, |
bool | pretty, | ||
int64_t | maxdecimals, | ||
int64_t | buffersize, | ||
const char * | nan_string = nullptr , |
||
const char * | infinity_string = nullptr , |
||
const char * | minus_infinity_string = nullptr , |
||
const char * | complex_real_string = nullptr , |
||
const char * | complex_imag_string = nullptr |
||
) | const |
Writes a JSON representation of this array to a destination
file.
destination | The file to write into. |
pretty | If true , add spacing to make the JSON human-readable. If false , return a compact representation. |
maxdecimals | Maximum number of decimals for floating-point numbers or -1 for no limit. |
buffersize | Size of a temporary buffer in bytes. |
nan_string | user-defined string for a not-a-number (NaN) value representation in JSON format |
infinity_string | user-defined string for a positive infinity representation in JSON format |
minus_infinity_string | user-defined string for a negative infinity representation in JSON format |
Although the JSON output contains all of the data from the array (and therefore might not be safe to print to the screen), it does not preserve the type information of an array. In particular, the distinction between ListType and RegularType is lost.
|
pure virtual |
Internal function to produce a JSON representation one node at a time.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
const std::string tostring | ( | ) | const |
Returns a string representation of this array (multi-line XML).
Although this XML string has detail about every node in the tree, it does not show all elements in the array buffers and therefore does not scale with the size of the dataset (i.e. it is safe to print to the screen).
Thus, it's also not a storage format: see tojson.
|
pure virtual |
Internal function to build an output string for tostring.
indent | Indentation depth as a string of spaces. |
pre | Prefix string, usually an opening XML tag. |
post | Postfix string, usually a closing XML tag and carriage return. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
High-level Type describing this array.
typestrs | A mapping from "__record__" parameters to string representations of those types, to override the derived strings. |
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns sorted unique values.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
pure virtual |
Returns an error message if this array is invalid; otherwise, returns an empty string.
Implemented in BitMaskedArray, ByteMaskedArray, EmptyArray, IndexedArrayOf< T, ISOPTION >, ListArrayOf< T >, ListOffsetArrayOf< T >, None, NumpyArray, Record, RecordArray, RegularArray, UnionArrayOf< T, I >, UnmaskedArray, and VirtualArray.
|
protected |
Internal function to determine if a particular set of parameters are allowed on a particular node (centralized by parameter, not by class).
|
protected |
See identities.
|
protected |
See parameters.