Understanding memory usage
The Xcalar architecture makes optimal use of system resources (such as memory, CPU, and network resources). Xcalar can optimize memory usage even on a heterogeneous cluster where server nodes have different amounts of memory installed.
About Xcalar Compute Engine's use of the memory hierarchy
Xcalar Compute Engine's distributed shared memory (DSM) architecture automatically uses and manages the tiers of a memory hierarchy, which are shown in the following list:
- Storage Class Memory (SCM)
- Solid state drives (SSDs)
- Hard disk drive (HDD)
Xcalar Compute Engine optimally distributes data and metadata across the cluster using sophisticated policies. When the Xcalar cluster is low on memory, Xcalar ages and serializes data to lower tiers of the memory hierarchy. It also deserializes data when necessary. While all of this is accomplished without user intervention, Xcalar recommends that users conserve memory (with methods described in Optimizing memory usage) so that Xcalar Compute Engine can deliver the highest possible performance.
Xcalar gives you fine-grained control over the distribution of data and metadata across the cluster. Such control is particularly useful when you have to deal with non-uniform data distribution or coloration that occurs for real-world applications. Production systems often need to run optimized dataflows in a similar way as database administrators optimize SQL queries. Contact Xcalar if you need Xcalar's Field Technology Engineers to help operationalize dataflows for you or provide training on operationalizing dataflows.
What happens when the cluster is low on memory
If Xcalar Compute Engine is low on memory, Xcalar Design displays a warning icon next to the user icon. A yellow icon indicates that memory consumption is at 80% and a red icon indicates that memory consumption is at 90%. A warning message is also displayed, recommending the action to take for releasing memory.
If the cluster is healthy without low-memory issues, is displayed next to the user icon.
Example of low-memory warning due to tables
The following partial screenshot illustrates the warning message when dropping tables is the recommended action.
Clicking the warning icon displays a modal window similar to the following for dropping tables:
The tables shown in the modal window are the same as the ones listed in the Tables panel that appears when you click the Worksheets icon. They are tables from all worksheets in your active workbook. See Optimizing memory usage for information about what you can do to release or conserve memory.
Example of low-memory warning due to datasets
The following partial screenshot illustrates the warning message when deleting datasets is the recommended action.
Clicking the warning icon displays the Datasets panel. For more information about dataset deletion, see Managing datasets.
Use the following list to determine which table to drop:
- From the Drop Tables modal window, you can see the amount of memory consumed by each table. The size information helps you decide the amount of memory that will be released due to the deletion.
Temporary tables are created when you perform operations in a worksheet to cleanse data, transform data, and so on. They can accumulate fast over time, and it is likely that you do not need most of the temporary tables. For example, it is likely that you no longer need the temporary table resulting from a data type change. For more information about temporary tables, see Understanding and changing table statuses.
The list of temporary tables might contain tables with .index appended to the table name (for example, airlines.index#CD08). These tables are created when certain operations take place. You can safely drop these tables.
- Hidden tables are the ones you removed from the worksheet to make the worksheet easier to manage. Re-evaluate whether the table will ever be used again. Drop the hidden tables that are no longer useful for modeling your queries. For more information about hidden tables, see Understanding and changing table statuses.
- You might have multiple active tables across worksheets in your workbook. It is least likely that you want to drop active tables because they are the ones that you currently work on or worked on recently. However, it is acceptable to drop an active table if it is not needed anymore, for example, because another active table contains similar data.
Dropping a table from the Drop Tables modal window
Follow these steps to drop tables:
- Click the Settings icon () on the toolbar to display the Settings menu.
- Click Release Memory to display the Drop Tables modal table.
In the Drop Tables modal, select the tables to drop. The following screenshot illustrates how to select the tables to drop. Before selecting a table, you can sort the tables by name, size, or date modified.
- Click DROP TABLES.