How to handle Vue 2 memory usage for large data (~50 000 objects)

Edit 20.4.2021 - Two years later, two years wiser

As this question/answer has gotten lots of attention and is still valid after all the years I wanted to throw few pointers. Most of the details underneath are is still valid. Still, I would direct towards using VueX with Lodash (or modern version of native JS functions) when dealing with filtered results & complex objects.

In order to ease the stress of your backend you can keep things simple: Fetch plain objects without related models. This means that your main results have only ID-keys to related objects. Use Axios or similar library to fetch all the related data with separate AJAX-requests ("customers", "projects", "locations") and use VueX to store 'em in their own list-properties. Create getters for each, such as:

projectsById: state => {    return _.keyBy(state.projects, "id")},

This way you can use related models for fetching labels and/or full objects when required and your backend doesn't need to fetch related data more than once. States and getters will be available within micro-components as well.

Basically: Avoid fetching full model-trees (even though C# EF or PHP Laravel provide tools for 'em) when dealing with large datasets. Use atomic approach: Fetch 20 different lists ("Axios.all([...])" is your friend!), each with own controller endpoint and cache the results to VueX store... And have fun ;)

Edit 12.03.2019 - additional tips at the end of this answer

It's been a while since I asked this question and I finally got to optimize this part of my project. I'd like to give few pointers for anyone having these performance and/or memory-issues.

Vue documentation never really explained it, but as Andrey pointed out you CAN use the component-object as an data-storage for your custom objects & object-lists. After all, it's just an normal javascript-object.

After optimization my list component setup looks somewhat like this:

module.exports = {    items: [],    mixins: [sharedUtils],    data: function() {        return {            columns: {                all: []    etc... Lot's of data & methods

The items-array is filled with thousands of complex objects (about 80mb of data, 6mb compressed) which I'm handling as non-reactive. This proved to be less of an issue than I would have thought -- Instead of using v-for directly against items I was already using structure in which I triggered filtering of this array whenever user clicked some filter-button and/or inputted string-filtering (such as name). Basically this "processFilters"-method goes through non-responsive items-array and returns filteredItems, which is stored in data-context. Thus it automatically becomes reactive as it's mutated.

<tr v-for="item in filteredItems" 

This way all the items within filteredItems stay reactive, but also lose reactivity when they are filtered out, thus saving bunch-load of memory. Whopping 1200mb shrunk to 400mb, which was exactly what I was looking for. Clever!

There are few issues which need to be addressed. Since items doesn't exist in data-context you cannot use it directly within template. This means that instead of writing...

<div v-if="items.length > 0 && everythingElseIsReady">

... I had to store length of items-array to separate data prop. This could have been fixed with computed value as well, but I like to keep those properties existing.

Giving up the reactivity of your main data-array isn't such a bad thing after all - The most important part is to understand that modifications which are made directly against items within that base-array are never triggering any changes to UI and/or sub-components (douh). This shouldn't be such an issue as long as you separate your code in such a way that you have "hidden data container" which holds all the results from backend, and you have smaller (filtered) presentation array of that large container. By using good REST-architecture you should already be good to go with non-reactive data-storage, as long as you remember to check that after saving the item within non-reactive data storage has also been updated to latest revision.

Additionally I was baffled by how little it matters performance-wise how many micro-components there are against hundreds of rows. The render takes a hit obviously, but even if I were to pass large props thousands of times (as I have thousands of instances of input-cells) it didn't seem to hit the memory. One of this kind of objects is my global translations-key/value-pair object, having over 20 000 lines of translated strings... but it still didn't matter. This makes sense, as Javascript uses object-references and Vue Core seems to be properly coded, so as long as you use such configuration objects as props you are simply referring from thousands of objects to the same data-set.

Finally, I'd say start going crazy with complex CRUD objects without fear of hitting memory limit!

Huge thanks for Andrey Popov for giving nudge towards right direction!

Tips (12.03.2019)

Since it's been a while and as I have continued building UI's with large & complex datasets I decided to drop few short ideas & tips.

1. Consider how you manage your master-records (ie. persons or products) vs related records (sub-objects / relational objects). Try to limit the amount of data injected for subcomponents, as you might be representing the same sub-object multiple times for different master-records. The problem is that it's possible that these objects are not actually reference-objects!

Consider situation where you have person-object, which contains city-object. Multiple persons live in the same city, but when you fetch JSON-data from backend are you sure are those duplicated city-objects actually one and same city (shared/referenced city-object between persons), or multiple representations of similar object (with data being exactly same, but under the hood each one being an individual instance / unique object). Let's say that you have 50 000 persons, each one containing the same sub-object/property "city": { id: 4, name: "Megatown" }, did you just fetch 50 000 individual city instances instead of just one? Is person1.city === person2.city , or do they just look the same and still be two different objects?

If you are unsure whether you are refering to shared city-object or using dozens of instances of similar sub-objects you could simply do there referencing inside your person-list-component. Your person contains city-id, so fetch list of cities with separate REST-method (getCities), and do the pairing on UI-level. This way you have only one list of cities, and you could resolve city from that that list and inject it to person, thus making reference to only one city. Alternatively you could resolve the city from list and pass it as an property to your person-component.

Also make sure to consider what is the purpose of the sub-object. Do you need it to be reactive, or is it static? In order to save bunch of memory you could just tell "person.city = city", which will be injected for each and every person-component, but if it needs to be reactive then you need to use Vue.set -method... and remember that if each city needs to be own instance (so that each person has similar city-object, but properties need to be editable per person) then you need to make sure that you are not using referred object! Thus you most likely need to clone the city-object, which will eat up browsers memory.

2. Your micro-component might contain separate view-states for both read-only-state and editor-state. This is quite common. Still, you are actually creating instance of that micro-component every time when, thus initializing that component thousands of times.

Think of situation where you have Excel-like spreadsheet with table and table-rows. Each cell contains your custom "my-input" -component, which takes "readonly"-property from your layout. If the UI is on the readonly-state then you are displaying only the label part inside that my-input-component, but otherwise you are displaying input-tag with some special conditions (such as having different input for datetime, number, text, textarea, select-tag etc). Now let's assume you have 100 rows with 20 columns, so you are actually initializing 2000 my-input-components. Now the question is -- what could be improved (performance-wise)?

Well, you could separate readonly-label from my-input-component to your list-view, so that you either display readonly-version (label) OR you you display the editable my-input-component. This way you have v-if condition, which makes sure that those 2000 micro-components won't be initialized unless you have specifically requested to initialize 'em (due either row or whole layout moving from readonly -> editable -state)... You probably guess how big the impact is memory-wise for browser, when Vue doesn't need to create 2000 components.

If you are facing that your page loads really slow it might not be VUE at all. Check out the amount of HTML-tags rendered to your HTML. HTML performs rather poorly when you have large amounts of tags. One of the simplest ways to demonstrate this is by repeating select-tag with 2000 options 100 times, or by having a single 20000 option select-tag. The same way you might be overflowing the amount of html-tags by having lots of micro-components with unnecessary wrapping divs etc... The less depth and less tags you have, the less rendering performance is required from browser & CPU.

Try to learn good HTML-tag architecture via examples. For an example you could study how Trello -services dashboard-view has been programmed. It's quite simple and beautiful representation of rather semi-complex service, with minimal amount of sub-divs.

There are many ways to improve memory handling, but I'd say that most important ones relate to separating "hidden" objects from visible objects, as described on my original answer. Second part is understanding the difference or instanced vs referenced objects. Third is to limit the amount of unnecessary data-passing between objects.

Personally I haven't tried this, but there exists a Vue-virtual-scroller component which handles any amount of data by simply being a wrapper for seemingly infinite amounts of data. Check out the concept @ https://github.com/Akryum/vue-virtual-scroller , and let me know if it solved the problem for you.

I hope these guidelines give some ideas for optimizing your components. Never give up the hope, there is always room for improvement!

From everything I've read, I see that you just don't need reactivity for that data, because:

Each row within table is toggleable, meaning that clicking the row changes the row to edit-mode, which enables Excel-like editing for that specific field/cell

This means rows are not editable and data cannot be mutated without user interaction.

Each object has about ~100-150 fields/properties, but only certain amount of 'em are shown at any given moment within table (table columns can be toggled in real-time).

You keep fields reactive but not display them.

And now your questions

Is there a way to toggle reactivity for specific array-list objects (by index or such), so that objects within array itself are unobserved/non-mutable unless specifically called to become mutable (ie. when user clicks row, which enables edit-mode)?

If there's a single item that can be edited at a time, then why keep everything reactive? You can easily use a single variable to listen for that changes.

How would you implement handling of large datasets for Vue, as reactivity seems to bottleneck the memory usage?

It's all about implementation - you rarely end up in a situation when you need a huge list of items to be reactive. The more items you have, the more events needs to happen in order to use the reactivity. If you have 50k items and there are just a few events to mutate (like user modifying data manually), then you can easily listen for those events and make the reactivity manually rather than leave Vue handle all the data. You can check Vuex that can make your life a bit easier for you :)

One idea I have is to turn that "items" -dataset to non-observable/non-reactive with Object.freeze or some similar approach and have table to render two datasets: one for non-reactive and one for those which are currently within edit-mode (which would be pushed to "editableItems" dataset when row is clicked)

This is kind of going in the right direction but there is no need to support two arrays. Imagine using something like this:

data: function() {    return {        editingItem: {}        // when editing is enabled bind the input fields to this item    }},created: function() {    this.items = [] // your items, can be used in markdown in the loop, but won't be reactive!},watch: {    editingItem: function(data) {        // this method will be called whenever user edits the input fields        // here you can do whatever you want        // like get item's id, find it in the array and update it's properties        // something like manual reactivity ;)    }}

• I had this exact problem where I needed to display a huge list, think 50000 items at least of variable height and I could not find any solution for it
• The general solution is to build/use a virtual scroll.
• It only keeps a few items in DOM while the rest of them are edited in the DOM. It however keeps changing what is visible depending on whether you scroll up/down
• The existing libraries I find do not deal with dynamic heights unless you HARDCODE the heights like vue-virtual-scroller and vue-virtual-scroll-list
• vue-collection-cluster allows you to dynamically calculate heights but lags miserably at 50000 items
• So I came up with my own solution that scrolls SUPER SMOOTH at 50000+ items, even tested with 100k items and works pretty well
• The idea of the implementation for dynamic row heights goes like this

• We need to maintain a list of heights for each item in an array

• Based on where the scroll Top is we apply a transform translateY vertically to offset the few items that we show the user at all times

• I have added ENOUGH comments in the solution for you to easily figure out what is going on

HTML

<script type="text/x-template" id="virtual-list">   <div id="root" ref="root">      <div id="viewport" ref="viewport" :style="viewportStyle">        <div id="spacer" ref="spacer" :style="spacerStyle">         <div v-for="i in visibleItems" :key="i.id" class="list-item" :ref="i.id" :data-index="i.index" @click="select(i.index)"  :class="i.index === selectedIndex ? 'selected': ''">           <div>{{ i.index + ' ' + i.value }}</div>   </div>   </div>   </div>   </div></script><div id="app">   <h1 class="title">      Vue.js Virtual + Infinite Scroll + Dynamic Row Heights + Arrow Key Navigation + No Libraries   </h1>   <p class="subtitle">      No hardcoding of heights necessary for each row. Set emitEnabled to false      for max performance. Tested with <span id="large_num">50000</span> items...   </p>   <div id="list_detail">      <div id="list">         <virtual-list></virtual-list>      </div>      <div id="detail">         <table>            <tbody>               <tr>                  <th class="caption">Root Container Height</th>                  <td>{{store['root-height']}} px</td>               </tr>               <tr>                  <th class="caption">Viewport Height</th>                  <td>{{store['viewport-height']}} px</td>               </tr>               <tr>                  <th class="caption">Smallest Row Height</th>                  <td>{{store['smallest-height']}} px</td>               </tr>               <tr>                  <th class="caption">Largest Row Height</th>                  <td>{{store['largest-height']}} px</td>               </tr>               <tr>                  <th class="caption">Scroll Top</th>                  <td>{{store['scroll-top']}} px</td>               </tr>               <tr>                  <th class="caption">Page Index</th>                  <td>{{store['page-start-index']}}</td>               </tr>               <tr>                  <th class="caption">Start Index</th>                  <td>{{store['start-index']}}</td>               </tr>               <tr>                  <th class="caption">End Index</th>                  <td>{{store['end-index']}}</td>               </tr>               <tr>                  <th class="caption">Translate Y</th>                  <td>{{store['translate-y']}} px</td>               </tr>            </tbody>         </table>         <p><b>Visible Item Indices on DOM</b> {{store['visible-items']}}</p>         <p><b>Total Height Till Current Page</b> {{store['page-positions']}}</p>         <p>            <b>Row's Vertical Displacement From Viewport Top</b>            {{store['row-positions']}}         </p>         <p><b>Heights</b> {{store['heights']}}</p>      </div>   </div></div>

CSS

@import url('https://fonts.googleapis.com/css?family=Open+Sans&display=swap');* {  margin: 0;  padding: 0;  box-sizing: border-box;}/**Apply Scroll Bar Styleshttps://css-tricks.com/the-current-state-of-styling-scrollbars/*/html {  --scrollbarBG: #181C25;  --thumbBG: orange;}body::-webkit-scrollbar {  width: 11px;}body {  scrollbar-width: thin;  scrollbar-color: var(--thumbBG) var(--scrollbarBG);}body::-webkit-scrollbar-track {  background: var(--scrollbarBG);}body::-webkit-scrollbar-thumb {  background-color: var(--thumbBG) ;  border-radius: 6px;  border: 3px solid var(--scrollbarBG);}html {  height: 100%;}body {  min-height: 100%;  height: 100%;  padding: 2rem;  color: #AAA;  background: #181C25;  font-family: 'Open Sans', sans-serif;  font-size: 0.9rem;  line-height: 1.75;}#app {  height: 100%;  display: flex;  flex-direction: column;}#list_detail {  display: flex;  height: 70%;}#list {  flex: 2;  height: 100%;}#detail {  flex: 1;  padding: 1rem;  overflow: auto;  height: 100%;}#root {  height: 100%;  overflow: auto;}.list-item {  padding: 0.75rem 0.25rem;  border-bottom: 1px solid rgba(255, 255, 0, 0.4);}.title {  color: white;  text-align: center;}.subtitle {  color: orange;  text-align: center;}table {  width: 100%;  table-layout: fixed;  text-align: center;}th.caption {  text-align: left;  color: #00BEF4;  font-weight: 100;  padding: 0.5rem 0;}td {  text-align: left;}b{  font-weight: 100;  color: #00BEF4;}#large_num {  color: red;}.selected {  background: midnightblue;}

Vue.js

I am getting limited to 30000 characters here on SO and therefore HERE is the complete code on CodePen

Limitations

• Does not play nice with screen resize at the moment, working on it

Features

• 50000+ items effortless scroll

• Arrow navigation supported just like a native list

• If you have any questions, let me know in the comments