Drill-XXXX: Refactor User Interface#3037
Draft
cgivre wants to merge 119 commits intoapache:masterfrom
Draft
Conversation
cgivre
changed the title
cgivre
added
enhancement
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/SqlLabSpaServlet.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/SqlLabSpaServlet.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DashboardResources.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DashboardResources.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/resources/webapp/src/pages/DashboardViewPage.tsx
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/resources/webapp/src/pages/DashboardViewPage.tsx
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/resources/webapp/src/pages/DashboardViewPage.tsx
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DashboardResources.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/resources/webapp/src/components/results/ResultsGrid.tsx
Fixed
Show fixed
Hide fixed
cgivre
force-pushed
the
feature/sqllab-react-ui
branch
from
080ba0b to
36af30f
Compare
exec/java-exec/src/main/resources/webapp/src/components/notebook/NotebookPanel.tsx
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/resources/webapp/src/components/notebook/NotebookPanel.tsx
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/ResultCacheService.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/ResultCacheService.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/ResultCacheService.java
Fixed
Show fixed
Hide fixed
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/ResultCacheService.java
Fixed
Show fixed
Hide fixed
cgivre
force-pushed
the
feature/sqllab-react-ui
branch
from
53da641 to
33fb68e
Compare
Root cause: executeQuery() was reusing the shared webUserConnection for multiple sequential queries (e.g. getTables then fetchColumnsForTable). The results.clear() call didn't properly clean up state, causing results from one query to contaminate the next query. Evidence from logs: - getTables: returns 4 rows (correct) - fetchColumnsForTable #1: Before clear=4 rows, After execute=0 rows (WRONG) - Second schema tree request: Same query returns 8 rows (correct) Solution: Create a fresh WebUserConnection for each executeQuery() call instead of reusing the shared connection. This ensures clean state and prevents cross-query contamination. Result: All metadata queries now return correct results consistently.
Use webSessionResources instead of non-existent methods.
…control Added useFreshConnection parameter to allow callers to choose: - true (default): Create fresh WebUserConnection to avoid state contamination (used for sequential metadata queries like in getSchemaTree) - false: Reuse shared webUserConnection with proper cleanup (available for performance-critical scenarios where safe) Overloaded executeQuery(sql) defaults to true for safety. Calls can use executeQuery(sql, false) to reuse the shared connection when they know it's safe and performance is critical. This gives flexibility: safe by default, but allows optimization when needed.
Three user-requested improvements: 1. Format suggested queries through SQL transpiler/formatter - Uses sql-formatter library to make queries clean and readable - Formatted queries shown in preview and applied to editor 2. Name new tabs with suggestion title - Modified addTab Redux action to accept optional tab name - QuerySuggestions now passes suggestion.title when creating tab - Tabs now show meaningful names like "Revenue by Product" instead of "Query N" 3. Remove expandable panel in modal - Removed Collapse wrapper from QuerySuggestions component - Content now displays directly with simple header - Cleaner UI, no unnecessary expansion/collapse Also updated QuerySuggestionsProps interface to pass both SQL and title.
Removed unused antd Space component import that was left over after removing the Collapse panel wrapper.
ESLint: allValidTables is only mutated (push), never reassigned, so it should be declared as const instead of let.
Updated onSelectSql callback to accept and forward the optional title parameter so that new tabs are named with the suggestion title.
- Add GeoJSON registry system (geoMapRegistry.ts) with 14 geographic maps grouped by region: World, North America, South America, Europe, Oceania, Asia - Move GeoJSON from static bundle to lazy-loaded public/geojson/ directory reduces initial bundle size by ~1MB - Implement async map registration with module-level Promise cache to prevent duplicate fetches during session - Add loading indicator while geographic data is fetched on first access - Fix Point geometry bug: replace us-states and ca-provinces centroids with actual Polygon boundaries by converting from TopoJSON with proper arc decoding - Expand map coverage from 3 maps (world, us-states, ca-provinces) to 14 maps including us-counties, mexican-states, brazilian-states, german-states, french-departments, uk-regions, australian-states, indian-states, etc. - Add Vite middleware to serve GeoJSON files in development environment - Update ColumnMapper with grouped geographic scope options Files modified: - ChartPreview: Remove static imports, add ensureMapRegistered() async function, add mapReady state with loading spinner, track mapId changes - ColumnMapper: Use grouped GeoJSON scope options from registry - vite.config.ts: Add middleware to serve /sqllab/geojson/ in dev mode - usStatesCaProvinces.ts: Add resolver functions for additional countries New files: - src/utils/geoMapRegistry.ts: Central geographic map registry - src/api/geojson.ts: Async GeoJSON fetching utility - public/geojson/: 13 GeoJSON files with proper Polygon geometry - scripts/download-geojson.py/sh: Data source documentation Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add effect to SqlLabPage that loads first saved query when opening a project Queries now appear in editor instead of blank tabs - Improve ProjectContext query retry logic to handle 404s correctly Don't retry 404 errors, only retry on network/server errors - Add exponential backoff for retries to reduce cache miss errors - Rename tab to match loaded saved query name for clarity
- Add us-zipcodes to geoMapRegistry with proper center/zoom for US - Create us-zipcodes.json GeoJSON file in public/geojson/ - Update download script to document ZCTA data source (Census Bureau) - Enables choropleth visualization at zip code level granularity
- Delete download-geojson.py and download-us-zipcodes.py (use shell script instead) - Fix ProjectContext: replace 'any' type with proper unknown type and type guard - Fix SqlLabPage: extract firstSavedQueryId variable for clean dependency tracking - All linting now passes: 0 errors, 0 warnings
- Remove dynamic import of savedQueries API to fix Vite chunking warning - Add static import of getSavedQuery at top of file - Eliminates 'dynamically imported but also statically imported' warning - No change to functionality, improves bundle optimization
- Delete incomplete us-zipcodes.json placeholder (only had 10 sample features) - Remove us-zipcodes from geoMapRegistry - Remove us-zipcodes section from download script Reason: Placeholder was causing blank screen when selected. Full US ZIP code GeoJSON data is very large (~100+ MB) and needs proper data source. Can be re-added when complete, validated data is available. Supported maps remain: world, us-states, us-counties, ca-provinces, mx-states, br-states, gb-regions, de-states, fr-departments, au-states, in-states, cn-provinces, jp-prefectures
- Add us-zipcodes.json with DC ZIP codes (working proof-of-concept) - Add us-zipcodes to geoMapRegistry for selection in choropleth maps - Update download script with instructions for full US data Current Status: ✓ DC ZIP codes working (172 features, ~1.2 MB GeoJSON) ✓ Builds successfully ✓ Linting passes Full US Data (optional upgrade): Users can get complete US ZCTA (all 50 states) by: 1. Installing GDAL: brew install gdal 2. Running the commands documented in download-geojson.sh 3. Replacing us-zipcodes.json with ~100 MB full dataset The lazy-loading architecture efficiently handles large map files, so full US data will work despite its size.
- Add .gitignore entries for large GeoJSON downloads (shapefiles, temp files) - Create setup-full-us-zipcodes.sh for users to download full US ZCTA data locally - Script uses GDAL (ogr2ogr) to convert Census Bureau shapefiles to GeoJSON - Full US data stays local only (not committed to git due to large size ~100 MB) - Current DC ZIP codes (~1.2 MB) remain committed as working foundation Usage: bash scripts/setup-full-us-zipcodes.sh Notes: - Requires GDAL: brew install gdal - Script handles download + format conversion automatically - Large files are ignored by git per .gitignore - Developers can have different local datasets without affecting version control
- Log when maps are fetched, registered, and re-used from cache - Log data points being rendered in choropleth with scope and map ID - Helps troubleshoot blank map issues when using geographic scopes - All logs use console.debug() to avoid noise in production Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…script for OpenDataDE data - Created process-zipcodes.py script that downloads all 50 state ZIP code files from OpenDataDE repo - Updated GeoMapRegistry with 50 state-level ZIP code entries (multiSelectAllowed: true) - Changed 'us-zipcodes' to 'US ZIP Codes (All)' for merged national data - State entries grouped under 'US ZIP Codes (by State)' for better UI organization - Each state entry includes proper center/zoom for that state's bounds - Script will create merged us-zipcodes.json (committed to git) + state files (git-ignored)
- Updated ColumnMapper to detect multi-select-enabled scopes (state ZIP codes) - Switch between single-select and multi-select UI dynamically - Store multi-selections in mapScopes array - Updated ChartPreview.tsx to handle both mapScope and mapScopes - When multiple state ZIP codes selected, automatically use all-states us-zipcodes map - Data filtering happens naturally - only selected states' ZIP codes will have values - Added debug logging showing number of selected states
The buildGroupedOptions function was missing 'US ZIP Codes (by State)' in the groups list, so the 50 state-level ZIP code options weren't appearing in the Geographic Scope selector dropdown. Now they'll be visible and selectable.
- Replaced old DC-only us-zipcodes.json with merged file from OpenDataDE - File contains 4,388 ZIP codes from successfully downloaded states - Size: 105.2 MB (committed to git for all users) - Individual state files excluded from git (can be regenerated locally) - Updated .gitignore to allow merged file but exclude individual state files
- Increase curl timeout from 30 seconds to 300 seconds (5 minutes) - All GeoJSON files minified (removes whitespace, ~50-70% size reduction) - Vite will auto-gzip for serving (~80-90% additional compression) - Better status messages during download - Should now successfully download all 50 states instead of timing out
- Implement coordinate reduction in simplify_geometry() - Keep every 3rd coordinate (reduces points by ~67%) - Maintains visual accuracy for choropleth maps - Combined with minification and gzip compression: - Individual state files: ~100-300 KB each (vs 3-13 MB before) - Merged file: ~10-15 MB (vs 105 MB before) - With gzip served by Vite: ~1-3 MB per state, ~2-5 MB merged
…rged all-states - Load individual state files (1-3 MB each) instead of merged file (100+ MB) - When user selects multiple states, show first state's map + filter data naturally - Reduces memory usage by 95% while maintaining full functionality - Users can visualize data from any state(s) with fast, responsive performance - Added UI hint recommending state-specific selection for best performance - If user has data from multiple states, filter query to those states first
- When metric values are NaN (undefined, null, or non-numeric), treat as 0 - Prevents missing data from breaking the visualization - Shows zero values on map for regions with no data instead of being blank
- Replace 384 MB us-zipcodes.json with 91 MB us-zipcodes.topojson - Enables GitHub compatibility (under 100 MB limit) - Add topojson-client package for client-side conversion - Update fetchGeoJson to detect and convert TopoJSON to GeoJSON for ECharts - Update .gitignore to track .topojson instead of .geojson - Update process-zipcodes.py to generate both formats - Successfully downloaded all 50 US states (32,936 total ZIP codes) Compression results: - Original GeoJSON with simplification: 384 MB - TopoJSON with mapshaper: 91 MB (76% reduction) - Individual state files: 1-26 MB each (not committed)
cgivre
force-pushed
the
feature/sqllab-react-ui
branch
from
33fb68e to
34bfc23
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRILL-XXXX: Refactor User Interface
Description
This PR refactors Drill's UI and refactors the Query view, adds visualizations and dashboards and in general makes Drill much more user friendly.
Documentation
(Please describe user-visible changes similar to what should appear in the Drill documentation.)
Testing
(Please describe how this PR has been tested.)