Advanced NumPy

Array container

Data type

In contrast to built-in Python containers (like lists) NumPy arrays can store elements of pre-determined type only. To see the type of array contents you can use the dtype attribute. Let’s look at two examples:

>>> a = np.array([1, 2, 3])
>>> a.dtype
dtype('int64')

>>> b = np.array([1., 2., 3.])
>>> b.dtype
dtype('float64')

In the first case the numbers are 64-bit (8-byte) integers and in the second 64-bit floating point (real) numbers. Note that NumPy auto-detects the data-type from the input. Specialised data types allow us to store data more compactly in memory, but most of the time we simply work with floating point numbers.

Note that all of the elements of an array must be of the same type. If we construct an array with different elements it will be cast to the “most general” type that can represent all elements. For example, array constructed from real numbers and integers will have a floating point data type:

>>> a = np.array([1., 2])
dtype('float64')

In case it is impossible, NumPy will use an object type (also represented by capital 'O') which can represent any Python object – even a function:

>>> def f(): pass
>>> a = np.array([f, f])
>>> a.dtype
dtype('O')

Some of NumPy features (like element-wise functions, np.abs, np.sqrt, etc., or reductions, np.sum, np.max etc.) won’t work with object arrays, but all types of indexing still work.

object type is most commonly encountered when constructing an array from multiple lists of different lengths:

>>> np.array([[1], [2, 3]])
array([[1], [2, 3]], dtype=object)

a = np.array([[1, 2], 
              [2, 3]])
b = np.array(['a', 'b', 'c'])
c = np.array([1, 2, 'a'])
d = np.array([np.dot, np.array])
e = np.random.randn(5) > 0
f = np.arange(5)

dtype = ?
np.array([('Bartosz', 5), ('Nelle', 4)], dtype=dtype)

Memory layout

NumPy array is just a memory block with extra information how to interpret its contents. Since memory has only linear address space, NumPy arrays need extra information how to lay out this block into multiple dimensions. This is done by means of shape and strides attributes:

Shape and strides

Lets try to reproduce this example. We first generate a 1D NumPy array of 8 elements:

>>> a = np.arange(8, dtype=np.uint8)
>>> a
array([0, 1, 2, 3, 4, 5, 6, 7], dtype=uint8)
>>> a.strides
(1,)
>>> a.shape
(8,)

shape and strides attributes are read-only, so we can not modify them directly. However, we my use as_strided function from NumPy library module:

>>> a1 = np.lib.stride_tricks.as_strided(a, strides=(4, 1), shape=(2,4))
>>> a1
array([[0, 1, 2, 3],
       [4, 5, 6, 7]], dtype=uint8)

Similarly, we can obtain the second example:

>>> a2 = np.lib.stride_tricks.as_strided(a, strides=(2, 1), shape=(3,4))
>>> a2
array([[0, 1, 2, 3],
       [2, 3, 4, 5],
       [4, 5, 6, 7]], dtype=uint8)

Note that in the second case the same data appears twice. However, it does not consume extra memory – all three arrays share the same memory block:

>>> a[2] = 100
>>> a1
array([[  0,   1, 100,   3],
       [  4,   5,   6,   7]], dtype=uint8)
>>> a2
array([[  0,   1, 100,   3],
       [100,   3,   4,   5],
       [  4,   5,   6,   7]], dtype=uint8)

def sliding_window(arr, size=2):
    """Produce an array of sliding window views of `arr`
    
    Parameters
    ----------
    arr : 1D array, shape (N,)
        The input array.
    size : int, optional
        The size of the sliding window.
        
    Returns
    -------
    arr_slide : 2D array, shape (N - size - 1, size)
        The sliding windows of size `size` of `arr`.
        
    Examples
    --------
    >>> a = np.array([0, 1, 2, 3])
    >>> sliding_window(a, 2)
    array([[0, 1],
           [1, 2],
           [2, 3]])
    """
    return arr  # fix this

x = np.random.rand(5, 10)
y = np.random.rand(10)
z = x + y

xb, yb = np.broadcast_arrays(x, y)