PyTorch - What does contiguous() do?

There are a few operations on Tensors in PyTorch that do not change the contents of a tensor, but change the way the data is organized. These operations include:

narrow(), view(), expand() and transpose()

For example: when you call transpose(), PyTorch doesn't generate a new tensor with a new layout, it just modifies meta information in the Tensor object so that the offset and stride describe the desired new shape. In this example, the transposed tensor and original tensor share the same memory:

x = torch.randn(3,2)y = torch.transpose(x, 0, 1)x[0, 0] = 42print(y[0,0])# prints 42

This is where the concept of contiguous comes in. In the example above, x is contiguous but y is not because its memory layout is different to that of a tensor of same shape made from scratch. Note that the word "contiguous" is a bit misleading because it's not that the content of the tensor is spread out around disconnected blocks of memory. Here bytes are still allocated in one block of memory but the order of the elements is different!

When you call contiguous(), it actually makes a copy of the tensor such that the order of its elements in memory is the same as if it had been created from scratch with the same data.

Normally you don't need to worry about this. You're generally safe to assume everything will work, and wait until you get a RuntimeError: input is not contiguous where PyTorch expects a contiguous tensor to add a call to contiguous().

From the pytorch documentation:

contiguous() → Tensor
Returns a contiguous tensor containing the same data as selftensor. If self tensor is contiguous, this function returns the selftensor.

Where contiguous here means not only contiguous in memory, but also in the same order in memory as the indices order: for example doing a transposition doesn't change the data in memory, it simply changes the map from indices to memory pointers, if you then apply contiguous() it will change the data in memory so that the map from indices to memory location is the canonical one.

tensor.contiguous() will create a copy of the tensor, and the element in the copy will be stored in the memory in a contiguous way.The contiguous() function is usually required when we first transpose() a tensor and then reshape (view) it. First, let's create a contiguous tensor:

aaa = torch.Tensor( [[1,2,3],[4,5,6]] )print(aaa.stride())print(aaa.is_contiguous())#(3,1)#True

The stride() return (3,1) means that: when moving along the first dimension by each step (row by row), we need to move 3 steps in the memory. When moving along the second dimension (column by column), we need to move 1 step in the memory. This indicates that the elements in the tensor are stored contiguously.

Now we try apply come functions to the tensor:

bbb = aaa.transpose(0,1)print(bbb.stride())print(bbb.is_contiguous())#(1, 3)#Falseccc = aaa.narrow(1,1,2)   ## equivalent to matrix slicing aaa[:,1:3]print(ccc.stride())print(ccc.is_contiguous())#(3, 1)#Falseddd = aaa.repeat(2,1)   # The first dimension repeat once, the second dimension repeat twiceprint(ddd.stride())print(ddd.is_contiguous())#(3, 1)#True## expand is different from repeat.## if a tensor has a shape [d1,d2,1], it can only be expanded using "expand(d1,d2,d3)", which## means the singleton dimension is repeated d3 timeseee = aaa.unsqueeze(2).expand(2,3,3)print(eee.stride())print(eee.is_contiguous())#(3, 1, 0)#Falsefff = aaa.unsqueeze(2).repeat(1,1,8).view(2,-1,2)print(fff.stride())print(fff.is_contiguous())#(24, 2, 1)#True

Ok, we can find that transpose(), narrow() and tensor slicing, and expand() will make the generated tensor not contiguous. Interestingly, repeat() and view() does not make it discontiguous. So now the question is: what happens if I use a discontiguous tensor?

The answer is it the view() function cannot be applied to a discontiguous tensor. This is probably because view() requires that the tensor to be contiguously stored so that it can do fast reshape in memory. e.g:

bbb.view(-1,3)

we will get the error:

---------------------------------------------------------------------------RuntimeError                              Traceback (most recent call last)<ipython-input-63-eec5319b0ac5> in <module>()----> 1 bbb.view(-1,3)RuntimeError: invalid argument 2: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Call .contiguous() before .view(). at /pytorch/aten/src/TH/generic/THTensor.cpp:203

To solve this, simply add contiguous() to a discontiguous tensor, to create contiguous copy and then apply view()

bbb.contiguous().view(-1,3)#tensor([[1., 4., 2.],        [5., 3., 6.]])