The normal upwards scrolling and the virtual scroll, both go hand in hand.

Using the rows with the class load-next-set and checking the direction in which the user is scrolling before it comes into view, gives a good idea of which set of rows need to be loaded, i.e. the ones above or the ones below. This could be done by adding the class to the 50th row from the top as well as the end.

To explain this better, for example, a user is scrolling down from the 101st row (Remember, the number of rows loaded at a given time is 100). Once the 150th row comes into view, the next set of rows i.e. 201st to 300th are appended into the table. So now you have rows from 100 to 300 present, with 150th and 250th rows with class load-next-set. If the user keeps going down, and the 250th row comes into view, we can safely remove the 101st to 200th row (and add 301st to 400th). Now if the user starts going upwards, and the 250th row is still in view or comes into view, we load the set above the present one, i.e. rows 101st to 200th are added to the top of the table again (and the 301st to 400th destroyed). So at any given time, there cannot be more than 200 rows present in the DOM.

For cases where a user might be going upwards and the 50th row from the top doesn’t come into view (for example, at the end of a particularly fast scroll, the 20th row from the top is visible and the user decides to go upwards), we look for whether the first row (the empty one added to push down the position of rows) has come into view (even partially) and load up the corresponding rows. Same for when we are going downwards.

So now, we will be keeping track of 4 element: the first and last load-next-set, the first-row and the last-row. The scroll event function with the changes looks like this:

var flag = 0;
$(table.parentNode.parentNode).bind('scroll', function(evt) {
  self._downwardDirection = self._scrollTop < $(this).scrollTop();

  var element = document.querySelectorAll('.load-next-set');
  element = element[element.length - 1];
  var position = element.getBoundingClientRect();
  var element2 = document.querySelector('.load-next-set');
  var position2 = element2.getBoundingClientRect();
  var lastElement = document.querySelector('.last-row');
  var positionLastElement = lastElement.getBoundingClientRect();
  var firstElement = document.querySelector('.first-row');
  var positionFirstElement = firstElement.getBoundingClientRect();

  if(self._downwardDirection && !flag) {
    if((position.top >= 0 && position.bottom <= window.innerHeight) || 
      (positionLastElement.top < window.innerHeight && positionLastElement.top > 0)) {
      console.log('Loading next set');
      flag = true;
      self._onBottomTable(self._scrollTop, table, this, evt);
    }
  }

  if(!self._downwardDirection && !flag) {
    if(position2.top >= 0 && position2.bottom <= window.innerHeight || 
      (positionFirstElement.bottom > 0 && positionFirstElement.bottom < window.innerHeight)) {
      console.log('Loading upper set');
      flag = true;
      self._onTopTable(self._scrollTop, table, this, evt);
    }
  }

  clearTimeout($.data(this, 'scrollTimer'));
  $.data(this, 'scrollTimer', setTimeout(function() {
    if(positionLastElement.top <= 0 && positionLastElement.bottom >= 0) {
      console.log('Last row is partially visible in screen');
      var goto = self.getPageNumberSrcolling(self._scrollTop);
      console.log(goto + ' ' + self._scrollTop);
      self._onChangeGotoScrolling(self._scrollTop, goto, table);
    }
      
    if(positionFirstElement.top < 0 && positionFirstElement.bottom >= window.innerHeight) {
      console.log('First row is partially visible in screen');
      var goto = self.getPageNumberSrcolling(self._scrollTop);
      console.log(goto + ' ' + self._scrollTop);
      self._onChangeGotoScrolling(self._scrollTop, goto, table);
    }
    flag = 0;
  }, 250));
  self._scrollTop = $(this).scrollTop();
});

So if the first load-next-set comes into view, and the user is scrolling upwards, we load the above set. Similarly, when the last load-next-set comes into view and the user is scrolling downwards, we load the below set. The flag is used to ensure that no loading of the upper set and next set occur concurrently (in cases where the first 50th row is also the last 50th row).

The onTopTable is similar to the onBottomtable with the start value set accordingly. In this step, we are keeping track of the first and last loaded row (present in the DOM at a time). pageStart refers to the first loaded row, and the totalSize refers to the last loaded row.

DataTableView.prototype._onTopTable = function(scrollPosition, table, elmt, evt) {
  if(this._pageStart - this._pageSize >= 0) {
    this._showRowsTop(scrollPosition, table, this._pageStart - this._pageSize);
  }
};

The if condition ensures that we don’t get into negative values while loading the upper set.

DataTableView.prototype._showRowsTop = function(scrollPosition, table, start, onDone) {
  console.log('_showRowsTop');
  var self = this;

  this._totalSize = start + 2 * this._pageSize;
  console.log(start);

  $('tr.load-next-set').removeClass('load-next-set');

  Refine.fetchRows(start, this._pageSize, function() {
    $('.last-row').remove();

    loadRows(start);
    self._adjustNextSetClasses(start, true);
    
    if (onDone) {
      onDone();
    }
  }, this._sorting);
};

The loadRows function receives an argument start to maintain the indices where each row is to be inserted (Note that showRowsBottom doesn’t pass this argument because we always insert rows to the bottom of the table).

The showRowsTop function sets the totalSize and calls the _adjustNextSetClasses with an additional argument true which indicates that the classes need to be adjusted according to the top set.

DataTableView.prototype._adjustNextSetClasses = function(start, top) {
  var heightToAddBottom = Math.max(0, this._sizeRowsTotal - (this._totalSize) * this._sizeRowFirst);

  if(top) {
    this._pageStart = start;
    heightToAddBottom = Math.max(0, this._sizeRowsTotal - (this._pageStart + 2 * this._pageSize) * this._sizeRowFirst);
  }

  if($('tbody tr').length > 102 && start > 100 && top == null) { // for going downwards
    console.log('Deleting above rows');
    $('tbody tr').slice(1, $('tbody tr').length - 2 * this._pageSize).remove();
    this._pageStart = start - this._pageSize;
  }
  if($('tbody tr').length > 201 && top) { // for going upwards
    console.log('Deleting below rows');
    $('tbody tr').slice(201, $('tbody tr').length).remove();
  }

  var heightToAddTop = (this._pageStart) * this._sizeRowFirst;
  $('tbody tr:first').css('height', heightToAddTop);

  document.querySelector('.data-table').insertRow();
  $('tr:last').css('height', heightToAddBottom);
  $('tr:last').addClass('last-row');

  if (theProject.rowModel.mode == "record-based") {
    $('tr.record').eq(-51).addClass('load-next-set');
    $('tr.record').eq(51).addClass('load-next-set');
  } else {
    $('tr').eq(-52).addClass('load-next-set');
    $('tr').eq(51).addClass('load-next-set');
  }
}

adjustNextSetClasses is where most of the work takes place: adding appropriate height to the first and last row in order to reflect the correct position and the deletion of rows that are not visible or required anymore.

Some of the calculations in the above code are a bit repetitive and will be updated in a cleaner manner in the PR.

Conclusion

The basic work for the virtual scrolling system is done. Now the main factor to focus on is the performance and optimization of these functions (and bugs and user feedback).