Article Type: Troubleshooting
Audience: Application Designers, Developers, Solution Architects
Module: Data Flows, Script Editor
Applies to Versions: Fuuz 2025.12+

1. Problem Overview

As industrial operations scale, applications must process increasingly large datasets for analytics, reporting, and operational intelligence. Developers often encounter performance concerns when transitioning from small test datasets to production-scale data volumes. This article demonstrates how properly optimized transforms scale efficiently with the Fuuz Data Flow engine, and provides guidance on handling datasets ranging from thousands to tens of thousands of records.

Symptoms

• Concern about scaling transforms from development to production data volumes
• Uncertainty about which transform technology (JSONata vs JavaScript) to use for large datasets
• Need to process 10,000+ records for aggregations, analytics, or reporting
• Desire to understand performance characteristics before production deployment

2. Performance Benchmarks

The following benchmarks compare transform performance across two real-world datasets using a workcenter history aggregation that performs complex grouping, duration calculations, and multi-dimensional analytics.

Dataset Specifications

Execution Time Results

* Estimated based on algorithmic complexity; not tested due to impractical execution times.

3. Understanding Scaling Behavior

Transform performance scaling depends on algorithmic complexity. Understanding these patterns helps predict how transforms will behave as data volumes grow.

Why JavaScript Scales Better

Scaling Patterns Explained

• O(n) - Linear: Time increases proportionally with data. Doubling records doubles time. Example: Single-pass aggregation
• O(n log n) - Log-linear: Slightly worse than linear due to sorting. Example: Sort then process
• O(n²) - Quadratic: Time increases with square of data. 10x records = 100x time. Example: Nested loops, $filter inside $map

The optimized JavaScript implementation achieves O(n log n) complexity—the sort operation is O(n log n), but all subsequent processing is O(n) single-pass. This explains why 6x more data results in only ~2x more time rather than 6x or 36x.

4. Using the Data Flow Designer

The Fuuz Data Flow Designer provides a visual environment for building and testing transforms with large datasets. JavaScript Transform nodes leverage the full power of the V8 engine for maximum performance.

Setting Up a Large Dataset Transform

1. Create a new Data Flow or open an existing flow
2. Add a Source Node to provide input data (GraphQL query, HTTP request, or static payload)
3. Add a JavaScript Transform Node connected to the source
4. Configure the JavaScript function with optimized aggregation logic
5. Test execution and monitor timing in the debug output

JavaScript Transform Node Configuration

Within the JavaScript Transform node, access input data via the $ variable. The transform function should return the processed result object.

// Access input data from the connected source node
const records = $.workcenterHistory;

// Perform optimized single-pass aggregation
const sorted = records.slice().sort((a, b) => {
  if (a.workcenterId < b.workcenterId) return -1;
  if (a.workcenterId > b.workcenterId) return 1;
  if (a.occurAt < b.occurAt) return -1;
  if (a.occurAt > b.occurAt) return 1;
  return 0;
});

// Build all aggregations in single pass
// ... (see full optimized script)

return {
  summary: { /* aggregated results */ },
  aggregations: { byMonth, byWeek, byDay }
};

5. Using the Script Editor

The Fuuz Script Editor provides an interactive environment for developing and testing transforms before deploying them to Data Flows. Both JSONata and JavaScript transforms can be tested with real payload data.

Testing Large Dataset Transforms

1. Open the Script Editor from the Fuuz application menu
2. Select JavaScript as the transform language (not JSONata for large datasets)
3. Paste or load the sample payload data into the input panel
4. Paste the optimized JavaScript function into the script panel
5. Execute the transform and observe execution time in the output

6. Optimization Techniques for Large Datasets

Apply these techniques to ensure transforms scale efficiently with data volume:

1. Single-Pass Multi-Aggregation

Build all groupings (by month, week, day, workcenter) in a single loop instead of multiple separate passes:

// EFFICIENT: Single pass builds all aggregations
for (let i = 0; i < records.length; i++) {
  const rec = records[i];
  
  // Update month aggregation
  if (!byMonthMap[rec.month]) byMonthMap[rec.month] = initAccumulator();
  byMonthMap[rec.month].total += rec.duration;
  
  // Update week aggregation (same loop)
  if (!byWeekMap[rec.yearWeek]) byWeekMap[rec.yearWeek] = initAccumulator();
  byWeekMap[rec.yearWeek].total += rec.duration;
  
  // Update day aggregation (same loop)
  if (!byDayMap[rec.day]) byDayMap[rec.day] = initAccumulator();
  byDayMap[rec.day].total += rec.duration;
}

2. Avoid Nested Filters

Never filter inside a loop—this creates O(n²) complexity:

// BAD - O(n²): Filters entire array for each record
records.forEach(rec => {
  const related = records.filter(r => r.workcenterId === rec.workcenterId);
});

// GOOD - O(n): Pre-group, then process
const byWorkcenter = {};
records.forEach(rec => {
  if (!byWorkcenter[rec.workcenterId]) byWorkcenter[rec.workcenterId] = [];
  byWorkcenter[rec.workcenterId].push(rec);
});

3. Use Fast String Comparisons

Replace localeCompare() with direct comparison operators for 2-3x faster sorting:

// SLOWER
records.sort((a, b) => a.occurAt.localeCompare(b.occurAt));

// FASTER (ISO date strings compare correctly with < >)
records.sort((a, b) => {
  if (a.occurAt < b.occurAt) return -1;
  if (a.occurAt > b.occurAt) return 1;
  return 0;
});

4. Cache Computed Values

Avoid recomputing the same values repeatedly:

// Cache year boundaries for week calculation
const jan1Cache = {};
function getJan1Ms(year) {
  if (!jan1Cache[year]) {
    jan1Cache[year] = Date.UTC(year, 0, 1);
  }
  return jan1Cache[year];
}

7. Capacity Planning Guidelines

Use these guidelines to plan for production workloads:

8. When to Escalate

Contact Fuuz Support if:

• Optimized JavaScript transforms take longer than expected based on capacity guidelines
• Data Flow execution fails with memory or timeout errors
• Performance degrades significantly after Fuuz platform updates
• You need guidance on architectural patterns for very large datasets (>100K records)
• Issue persists after implementing all optimization recommendations

9. Related Resources

Sample Files

• Fuuz_Flow_Large_Dataset_Javascript_0_0_1.json - Importable Data Flow with 14,368 record dataset and JavaScript transform
• payload_workcenter_history_medium.json - Medium test dataset (14,368 records, 20 workcenters, 3 months)
• workcenter_history_aggregation_payload query - Small test dataset (2,343 records, 5 workcenters, 1 month)
• workcenter_history_aggregation_javascript_v1 - Optimized JavaScript aggregation function

Related Articles

• Troubleshooting Slow Transform Performance: JSONata vs JavaScript Optimization Guide
• Data Flow Designer: Getting Started
• JavaScript Transform Node Reference

10. Revision History