Article Type: Troubleshooting
Audience: Application Designers, Developers, Solution Architects
Module: Data Flows, Script Editor
Applies to Versions: Fuuz 3.0+

1. Problem Overview

Data transformation operations in Fuuz can experience significant performance degradation when processing large datasets. This guide helps developers identify performance bottlenecks in JSONata transforms and provides optimization strategies, including migration to JavaScript when necessary.

Symptoms

• Transform execution takes more than 10 seconds for datasets under 5,000 records
• Script Editor becomes unresponsive during transform execution
• Data Flow nodes timeout or fail with large payloads
• Execution time increases exponentially as record count grows (not linearly)
• Browser memory usage spikes during transform processing

2. Performance Benchmarks

The following benchmarks demonstrate performance differences between implementation approaches, tested with a workcenter history aggregation transform processing 2,343 records with complex grouping, duration calculations, and multi-level aggregations.

3. Common Issues

4. Detailed Resolution Steps

Solution 1: Identify Performance Bottlenecks

Before optimizing, identify the root cause of slow performance:

1. Check for O(n²) patterns: Search for $filter or $map nested inside another $map or loop
2. Count $toMillis calls: Each date parsing operation is expensive; minimize redundant conversions
3. Look for repeated filtering: Same filter condition applied multiple times in different sections
4. Test with smaller datasets: If doubling records more than doubles execution time, you have an O(n²) problem

Example O(n²) Anti-Pattern:

/* SLOW - O(n²): For each record, filters entire array */
$map($records, function($rec, $idx) {(
  $nextRec := $filter($records, function($r, $i) {
    $i > $idx and $r.workcenterId = $rec.workcenterId
  })[0];
  /* ... */
)})

Optimized O(n) Solution:

/* FAST - O(n): Sort once, use index arithmetic */
$sorted := $records^(workcenterId, occurAt);
$len := $count($sorted);

$map($sorted, function($rec, $idx) {(
  $next := $idx < $len - 1 ? $sorted[$idx + 1] : null;
  $sameWC := $next != null and $next.workcenterId = $rec.workcenterId;
  /* ... */
)})

Solution 2: JSONata Optimization Techniques

Apply these optimizations to improve JSONata transform performance:

1. Pre-sort and use index arithmetic for lookups:

/* Sort once at the beginning */
$sorted := workcenterHistory^(workcenterId, occurAt);
$len := $count($sorted);

/* Use index to access next record - O(1) instead of O(n) */
$enriched := $sorted#$idx.(
  $next := $idx < $len - 1 ? $sorted[$idx + 1] : null;
  /* Process record */
)

2. Pre-filter data once, reuse throughout:

/* Filter once at the start */
$downtimeRecords := $enriched[modeId = "disabled"];
$prodRecs := $enriched[modeId = "Production"];
$idleRecs := $enriched[modeId = "Idle"];

/* Reuse filtered arrays throughout transform */
$dtTotal := $sum($downtimeRecords.calculatedDuration);

3. Cache expensive calculations:

/* Cache milliseconds value */
$ms := $toMillis(occurAt);
$yr := $substring(occurAt, 0, 4);
$jan1ms := $toMillis($yr & "-01-01");

/* Reuse cached values */
$dayOfYear := $floor(($ms - $jan1ms) / 86400000) + 1;

4. Use $merge with $each for grouping operations:

/* Efficient grouping with $each */
$byMonth := $each($enriched{month: $}, function($recs, $period) {
  $merge([{"period": $period}, $aggStats($recs)])
})^(period);

Solution 3: Migrate to JavaScript for Maximum Performance

When JSONata optimization is insufficient, migrate to a JavaScript Script Node:

JavaScript Script Node Syntax:

/* Access input data via $ variable */
const records = $.workcenterHistory;

/* Use native JavaScript array methods */
const sorted = [...records].sort((a, b) => {
  const wcCompare = a.workcenterId.localeCompare(b.workcenterId);
  return wcCompare !== 0 ? wcCompare : a.occurAt.localeCompare(b.occurAt);
});

/* Process with forEach, map, reduce */
sorted.forEach((rec, idx) => {
  const next = idx < sorted.length - 1 ? sorted[idx + 1] : null;
  // Process record
});

/* Return result object */
return {
  summary: { /* ... */ },
  aggregations: { /* ... */ }
};

5. Technology Selection Guide

Use this guide to select the appropriate technology for your transform:

Decision Flowchart

1. Start with JSONata for initial development and small datasets
2. If performance is slow: Apply JSONata optimization techniques (Solution 2)
3. If still slow after optimization: Check for O(n²) patterns and fix them
4. If optimized JSONata exceeds 15-20 seconds: Migrate to JavaScript
5. For production with large datasets: Always use JavaScript or pre-aggregate data upstream

6. Prevention & Best Practices

JSONata Best Practices

• Sort once, process sequentially: Use ^(field1, field2) at the start and index arithmetic for lookups
• Pre-filter data: Create filtered arrays once and reuse them throughout the transform
• Avoid nested $filter inside $map: This creates O(n²) complexity
• Cache expensive calculations: Store $toMillis() results in variables
• Use $merge carefully: In some Fuuz versions, direct object construction may be faster
• Test with production-sized datasets: Performance issues often only appear at scale

JavaScript Best Practices

• Use native array methods: forEach, map, reduce, filter are highly optimized
• Prefer objects for grouping: Use objects as hash maps for O(1) key lookup
• Use Set for unique values: new Set(array.map(x => x.field)).size for counting distinct values
• Avoid creating objects in tight loops: Reuse objects where possible
• Access input via $: The Fuuz input variable is $, not payload

Architecture Recommendations

• Pre-aggregate in Data Flows: Use upstream nodes to reduce data volume before complex transforms
• Filter by date range: Process only the time period needed, not entire history
• Split complex transforms: Break into multiple simpler transforms that can run in parallel
• Use GraphQL aggregation: Push aggregation to the database layer when possible
• Cache results: Store computed aggregations if data doesn't change frequently

7. When to Escalate

Contact Fuuz Support if:

• JavaScript Script Node produces syntax errors that appear to be parser bugs
• Transform performance degrades significantly between Fuuz versions
• Memory errors or crashes occur during transform execution
• Optimized transforms still timeout in production Data Flows
• Issue persists after implementing all optimization recommendations

When escalating, provide:

• Transform code (JSONata or JavaScript)
• Sample input data (anonymized if necessary)
• Record count and execution time measurements
• Fuuz version and environment (Build, QA, Production)
• Any error messages or screenshots

8. Related Resources

Sample Scripts

The following sample scripts demonstrate the optimization techniques described in this article:

• workcenter_history_aggregation_jsonata_v1- Original JSONata transform (baseline)
• workcenter_history_aggregation_jsonata_v2 - Optimized JSONata with O(n) algorithms
• workcenter_history_aggregation_javascript_v1 - JavaScript implementation for maximum performance
  Try these out for yourself in the Fuuz Script Editor!

Documentation Links

• JSONata Documentation - Official JSONata language reference
• Fuuz Platform Documentation - Data Flow and Script Node guides

9. Revision History