christian
Repositories Public Activity Starred Repositories
Loading Heatmap…

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • a6a457bbc9 Fix Cube Store workers sharing remote dir with router Workers had isolated CUBESTORE_REMOTE_DIR volumes, causing cleanup errors because they couldn't see the router's files. Now all workers mount the router's data directory as shared remote storage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • e03edeb79c Move dropPreAggregationsWithoutTouch to env var (not supported as config option) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • b205778d26 Switch pre-aggregation refresh from time-based to data-driven and enable auto-cleanup Replace `every: '1 hour'` with UPDATE_TIME-based refresh keys across 10 cubes to prevent unnecessary pre-aggregation table accumulation. Tables are now only rebuilt when source data actually changes. Enable dropPreAggregationsWithoutTouch for automatic cleanup of unused pre-aggregation tables. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
  • 4d5ee58a82 Merge remote branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
  • 778de9f937 Fixed git pull in cleanUp file
  • f36a253e85 Add pre-flight checks to deployment docs Document schema compatibility checks needed before running stored procedures: missing columns (gucci_approved) and duplicate gateway subscription entries that cause PRIMARY KEY and Data too long errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
  • Compare 4 commits »

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 9729072d03 Disable Activity.used_for_brand filter that dropped ~800 suppliers The queryRewrite filter (added in 2b302b8) acted as a WHERE clause, removing suppliers without identification_new records for the brand. The original LEFT JOIN condition kept those rows with NULL values. BrandSuppliers already filters by brand_id in its SQL, making this filter unnecessary for the BrandSuppliers+Activity query path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • e4d6cd1ec7 Fix compliance criteria mapping for non-Formulas cubes The boolean mapping (is_zdhc_crowdsource etc.) was applied to all queries, but these columns only exist in formula_cube. Non-Formulas cubes like AldiMonthlyReport, CandA, BrandActivityWithCompliance use SECURITY_CONTEXT.COMPLIANCE_CRITERIA in their SQL and need the original column names (gots, toxfmd, etc.). Now the mapping is split: Formulas queries get boolean columns, all others get original SQL. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • de2ba0bd75 Fix ActualConnections not filtered by ORGANIZATION_ID ActualConnections was incorrectly added to the queryRewrite exclusion list during refactoring, preventing the Factories.organization_id filter from being applied. Factory users saw all connections instead of only their own. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • db42babbea Remove nginx caching layer (files and documentation) The nginx njs-based cache had unresolved issues with POST body handling and CORS. Removed nginx/ directory and all nginx cache references from performance-refactoring.md and phase2-optimization.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • Compare 2 commits »

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • b537dc16f6 Simplify nginx cache: remove POST interception that broke large requests The error_page 418 + return 418 pattern discarded the POST body, causing "network connection interrupted" errors in Angular for queries with large filter lists. POST now passes through uncached to Cube.js (proxy_cache_methods defaults to GET HEAD). Only GET is cached. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 72afca4f33 Fix nginx cache to handle GET requests from Angular client The Angular Cube.js client sends queries as GET with URL-encoded params, not POST. Added GET support using js_set + proxy_cache_key with $args, and route POST to njs handler via error_page 418 pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 2710930aaf Fix nginx cache: use njs js_content + subrequest for reliable POST body caching $request_body is not available in proxy_cache_key because nginx evaluates the key before reading the POST body, causing all queries to share the same cache entry. New approach: njs js_content handler reads the full body via r.requestText, computes a hash (JWT fields + body hash), and makes a subrequest to an internal location that uses the hash as proxy_cache_key. Removes broken CORS handling (Cube.js sets its own headers). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 03b725e5c6 Add nginx response cache and phase 2 optimization plan nginx: JWT-aware caching layer for POST /cubejs-api/v1/load using njs module to extract BRAND_ID, COMPLIANCE_CRITERIA, SCOPE, USED_FOR_BRAND, ORGANIZATION_ID from JWT payload as cache key. Users of the same brand or factory share cache entries (USER_ID excluded from key). 2 min TTL with proxy_cache_lock to prevent thundering herd. Docs: phase 2 optimization plan covering MySQL indexes, BrandSuppliers refactoring, query cache tuning, and nginx cache deployment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 6f9322d472 Add multi-user load simulation to benchmark tool and missing formula_cube indexes Benchmark: multi-user mode with concurrent sessions (tokens × users), per-query progress tracking, Avg/Min/Max/P95/Errors table, summary cards, and Top 10 chart with Min/Max whiskers. Single-user mode unchanged. SQL: add missing MySQL indexes on formula_cube (cube_type, organization_id, scan_date, formula_id, basic_chemical, chemical_identifier) to fix full table scans in stored procedures and pre-aggregation builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • c573581606 Enhance benchmark tool and extend performance documentation Benchmark: add result comparison mode, formula ID filter, wall-clock timing for parallel execution, non-scrollable table for PDF export, and page title in PNG export. Docs: add sections for formula_cube procedures, pre-aggregation reset, and benchmark tool usage instructions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • Compare 2 commits »

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 42d2aef600 Add performance benchmark tool for side-by-side Cube.js server comparison Self-contained HTML app with 51 queries from warmUp.sh, supporting sequential/parallel execution, results table, SVG chart, and PNG/PDF export. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • e2c6e1287c Add warm-up script for pre-aggregation and query cache priming 50 queries across 5 phases covering all 20 cubes with pre-aggregations. Includes progress bar, dynamic quarter date range, and simplified versions of large-ID-filter queries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • Compare 2 commits »

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 27fd5b79c7 Add performance refactoring documentation (MD + PDF) Documents problems, solutions, deployment sequence, and best practices for Cube.js pre-aggregations and compliance booleans. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • a388535ee9 Fix container name in monitor script Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • a5f4443bcb Add script to monitor pre-aggregation builds Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • f5f6b9b39e Export insertIntoFormulaCube procedure with compliance boolean computation Same procedure as on staging, computing all 12 compliance boolean columns after inserting new data from formula_recent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 98fe86ca2e Switch formula-related cubes to data-driven refreshKey Instead of time-based refresh (every 1-2 hours), check formula_cube UPDATE_TIME every 10 minutes. Pre-aggregations only rebuild when insertIntoFormulaCube has actually finished updating the data. Affected cubes: Formulas, FormulaBrandUsage, ChemIq, ProductOrderChemicals Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2 months ago

christian pushed to main at dataIntegration/bhive-cubejs-refactoring

  • 8094068662 Add one-time compliance boolean computation to migration Computes all 12 compliance columns for existing formula_cube rows so values are correct immediately after adding the columns. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • 2b2bac35ab Add migration to create compliance boolean columns on production Adds 12 compliance columns to formula_cube and formula_old tables in cube_datasource. Must run before deploying new procedures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
  • Compare 2 commits »

2 months ago