Does subsetting (matrices or arrays) always perform a partial copy?

3 years ago by stravanasu to c/r_programming

Some large datasets are pushing memory and some functions I'm writing to the limit. I wanted to ask some questions about subsetting, of matrices and arrays in particular:

  1. Does defining a variable as a subset of another lead to copy? For instance
x <- matrix(rnorm(20*30), nrow=20, ncol=30)
y <- x[, 1:10]

Some exploration with object_size from pryr seems to indicate that a copy is made when y is created, but I'd like to be sure.

  1. If I enter a subset of a matrix/array as argument to a function, does it get copied before the function is started? For instance in
x <- matrix(rnorm(20*30), nrow=20, ncol=30)
y <- dnorm(0, mean=x[,1:10], sd=1)

I wonder if the data in x[,1:10] are copied and then given as input to dnorm.

I've heard that data.table allows one to work with subsets without copies being made (unless necessary), but it seems that one is constrained to two dimensions only – no arrays – that way.

Cheers!

MalditoBarbudo 3 points 3 years ago

As far as I know, yeah, R always work with copy on modification. Some libraries as you mention (data.table) can have object/classes to avoid this, but I'm not aware of any of them working with arrays (more than 2D). Maybe parquet or arrow have something like this??

path: 0 1430839, hotness: undefined, score: 3, children: 2
pglpm 4 points 3 years ago

Thank you for the suggestion! Worth looking at parquet and arrow indeed.

path: 0 1430839 1432094, hotness: undefined, score: 4, children: 0
morcution 2 points 3 years ago

+1 for parquet and arrow. If you're pushing memory better to just treat it as a completely out of memory problem. If you can split the data into multiple parquet files with hive style or directory partitioning it will be more efficient. You don't want parquet files too small though (I've heard people saying 1 GB each file is ideal, colleagues at work like 512 MB per file - but that's on an AWS setup).

Bonus is once you've learned the packages it'll be the same for all out of memory big datasets.

path: 0 1430839 1574612, hotness: undefined, score: 2, children: 0
MalditoBarbudo 0 points 3 years ago
path: 0 1453061, hotness: undefined, score: 0, children: 0
r_programming
r_programming

@programming.dev

login for more options
314
44
1

Please use this as a forum to discuss R, and learn more about it. If you have any questions about how to do specific things in R, this is the place to ask.

Getting Started

You can download R here.

You can download RStudio here. RStudio IDE, which is supported by Posit PBC, is a powerful and well-developed IDE for R. Other development environment options include Emacs addon Emacs Speak Statistics and VSCode.

Other Communities

Other communities that may be of interest across the fediverse:

Please send @a_statistician a message to recommend additional communities to add to this list.

Learning resources:

  • R for Data Science - a good introductory book for learning R. Start here if you're overwhelmed.
  • Big Book of R - collection of more than 500 online books/tutorials covering various aspects of R. Some links are to paid books with previews, but most links are to free online textbooks.

go to feed...