Understanding memory usage

The Xcalar architecture makes optimal use of system resources (such as memory, CPU, and network resources). Xcalar can optimize memory usage even on a heterogeneous cluster where server nodes have different amounts of memory installed.

About Xcalar Compute Engine's use of the memory hierarchy

Xcalar Compute Engine's distributed shared memory (DSM) architecture automatically uses and manages the tiers of a memory hierarchy, which are shown in the following list:

  • DRAM
  • Storage Class Memory (SCM)
  • Solid state drives (SSDs)
  • Hard disk drive (HDD)

Xcalar Compute Engine optimally distributes data and metadata across the cluster using sophisticated policies. When the Xcalar cluster is low on memory, Xcalar ages and serializes data to lower tiers of the memory hierarchy. It also deserializes data when necessary. While all of this is accomplished without user intervention, Xcalar recommends that users conserve memory (with methods described in Optimizing memory usage) so that Xcalar Compute Engine can deliver the highest possible performance.

Xcalar gives you fine-grained control over the distribution of data and metadata across the cluster. Such control is particularly useful when you have to deal with non-uniform data distribution or coloration that occurs for real-world applications. Production systems often need to run optimized dataflows in a similar way as database administrators optimize SQL queries. Contact Xcalar if you need Xcalar's Field Technology Engineers to help operationalize dataflows for you or provide training on operationalizing dataflows.

What happens when the cluster is low on memory

If Xcalar Compute Engine is low on memory, Xcalar Design displays a warning icon next to the user icon. A yellow icon indicates that memory consumption is at 80% and a red icon indicates that memory consumption is at 90%. A warning message is also displayed, recommending the action to take for releasing memory. 

If the cluster is healthy without low-memory issues, is displayed next to the user icon.

TIP: You can mouse over to display the percentage of memory used. Click the icon to display the Statistics window, where you can see more details about resource usage.

Example of low-memory warning due to tables

The following partial screenshot illustrates the warning message when dropping tables is the recommended action.

Clicking the warning icon displays a modal window similar to the following for dropping tables:

The tables shown in the modal window are the same as the ones listed in the Tables panel that appears when you click the Worksheets icon. They are tables from all worksheets in your active workbook. See Optimizing memory usage for information about what you can do to release or conserve memory.

NOTE: You can display the Drop Table modal window at any time. Click the Settings icon under Monitor. Then in the Settings panel, click Release Memory. For more information about the monitor, see Using the Monitor.

Example of low-memory warning due to datasets

The following partial screenshot illustrates the warning message when deleting datasets is the recommended action.

Clicking the warning icon displays the Datasets panel. For more information about dataset deletion, see Managing datasets.

Determining which table to drop

Use the following list to determine which table to drop:

  • From the Drop Tables modal window, you can see the amount of memory consumed by each table. The size information helps you decide the amount of memory that will be released due to the deletion.
  • Temporary tables are created when you perform operations in a worksheet to cleanse data, transform data, and so on. They can accumulate fast over time, and it is likely that you do not need most of the temporary tables. For example, it is likely that you no longer need the temporary table resulting from a data type change. For more information about temporary tables, see Understanding and changing table statuses.

    The list of temporary tables might contain tables with .index appended to the table name (for example, airlines.index#CD08). These tables are created when certain operations take place. You can safely drop these tables.

  • Hidden tables are the ones you removed from the worksheet to make the worksheet easier to manage. Re-evaluate whether the table will ever be used again. Drop the hidden tables that are no longer useful for modeling your queries. For more information about hidden tables, see Understanding and changing table statuses.
  • You might have multiple active tables across worksheets in your workbook. It is least likely that you want to drop active tables because they are the ones that you currently work on or worked on recently. However, it is acceptable to drop an active table if it is not needed anymore, for example, because another active table contains similar data.
IMPORTANT: Dropping any kind of table is not reversible. You cannot undo the action of dropping a table.

Dropping a table from the Drop Tables modal window

Follow these steps to drop tables:

  1. Click the Settings icon () on the toolbar to display the Settings menu.
  2. Click Release Memory to display the Drop Tables modal table.
  3. In the Drop Tables modal, select the tables to drop. The following screenshot illustrates how to select the tables to drop. Before selecting a table, you can sort the tables by name, size, or date modified.

  4. Click DROP TABLES.

Go to top