Update README.md

README.md

@@ -1,8 +1,9 @@
 # FHIRFLARE IG Toolkit
+![FHIRFLARE Logo](static/FHIRFLARE.png)
 
 ## Overview
 
-The FHIRFLARE IG Toolkit is a Flask-based web application designed to streamline the management, processing, validation, and deployment of FHIR Implementation Guides (IGs) and test data. It offers a user-friendly interface for importing IG packages, extracting metadata, validating FHIR resources or bundles, pushing IGs to FHIR servers, converting FHIR resources to FHIR Shorthand (FSH), and uploading complex test data sets with dependency management. The toolkit includes live consoles for real-time feedback, making it an essential tool for FHIR developers and implementers.
+The FHIRFLARE IG Toolkit is a Flask-based web application designed to streamline the management, processing, validation, and deployment of FHIR Implementation Guides (IGs) and test data. It offers a user-friendly interface for importing IG packages, extracting metadata, validating FHIR resources or bundles, pushing IGs to FHIR servers, converting FHIR resources to FHIR Shorthand (FSH), uploading complex test data sets with dependency management, and retrieving/splitting FHIR bundles. The toolkit includes live consoles for real-time feedback, making it an essential tool for FHIR developers and implementers.
 
 The application can run in two modes:
 
@@ -30,6 +31,12 @@ This toolkit offers two primary installation modes to suit different needs:
 ## Features
 
 * **Import IGs:** Download FHIR IG packages and dependencies from a package registry, supporting flexible version formats (e.g., `1.2.3`, `1.1.0-preview`, `current`) and dependency pulling modes (Recursive, Patch Canonical, Tree Shaking).
+* **Enhanced Package Search and Import:**
+    * Interactive page (`/search-and-import`) to search for FHIR IG packages from configured registries.
+    * Displays package details, version history, dependencies, and dependents.
+    * Utilizes a local database cache (`CachedPackage`) for faster subsequent searches.
+    * Background task to refresh the package cache from registries (`/api/refresh-cache-task`).
+    * Direct import from search results.
 * **Manage IGs:** View, process, unload, or delete downloaded IGs, with duplicate detection and resolution.
 * **Process IGs:** Extract resource types, profiles, must-support elements, examples, and profile relationships (`structuredefinition-compliesWithProfile` and `structuredefinition-imposeProfile`).
 * **Validate FHIR Resources/Bundles:** Validate single FHIR resources or bundles against selected IGs, with detailed error and warning reports (alpha feature). *Note: Lite version uses local SD checks only.*
@@ -55,9 +62,19 @@ This toolkit offers two primary installation modes to suit different needs:
     * Handles large numbers of files using a custom form parser.
 * **Profile Relationships:** Display and validate `compliesWithProfile` and `imposeProfile` extensions in the UI (configurable).
 * **FSH Converter:** Convert FHIR JSON/XML resources to FHIR Shorthand (FSH) using GoFSH, with advanced options (Package context, Output styles, Log levels, FHIR versions, Fishing Trip, Dependencies, Indentation, Meta Profile handling, Alias File, No Alias). Includes a waiting spinner.
+* **Retrieve and Split Bundles:**
+    * Retrieve specified resource types as bundles from a FHIR server.
+    * Optionally fetch referenced resources, either individually or as full bundles for each referenced type.
+    * Split uploaded ZIP files containing bundles into individual resource JSON files.
+    * Download retrieved/split resources as a ZIP archive.
+    * Streaming progress log via the UI for retrieval operations.
 * **FHIR Interaction UIs:** Explore FHIR server capabilities and interact with resources using the "FHIR API Explorer" (simple GET/POST/PUT/DELETE) and "FHIR UI Operations" (Swagger-like interface based on CapabilityStatement). *Note: Lite version requires custom server URLs.*
-* **API Support:** RESTful API endpoints for importing, pushing, retrieving metadata, validating, and uploading test data.
+* **HAPI FHIR Configuration (Standalone Mode):**
-* **Live Console:** Real-time logs for push, validation, upload test data, and FSH conversion operations.
+    * A dedicated page (`/config-hapi`) to view and edit the `application.yaml` configuration for the embedded HAPI FHIR server.
+    * Allows modification of HAPI FHIR properties directly from the UI.
+    * Option to restart the HAPI FHIR server (Tomcat) to apply changes.
+* **API Support:** RESTful API endpoints for importing, pushing, retrieving metadata, validating, uploading test data, and retrieving/splitting bundles.
+* **Live Console:** Real-time logs for push, validation, upload test data, FSH conversion, and bundle retrieval operations.
 * **Configurable Behavior:** Control validation modes, display options via `app.config`.
 * **Theming:** Supports light and dark modes.
 
@@ -101,9 +118,10 @@ docker run -d \
   -v ./logs:/app/logs \
   --name fhirflare-lite \
   ghcr.io/sudo-jhare/fhirflare-ig-toolkit-lite:latest
-
 Standalone Version (Includes local HAPI FHIR):
 
+Bash
+
 # Pull the latest Standalone image
 docker pull ghcr.io/sudo-jhare/fhirflare-ig-toolkit-standalone:latest
 
@@ -119,7 +137,6 @@ docker run -d \
   -v ./logs:/app/logs \
   --name fhirflare-standalone \
   ghcr.io/sudo-jhare/fhirflare-ig-toolkit-standalone:latest
-
 Building from Source (Developers)
 Using Windows .bat Scripts (Standalone Version Only):
 
@@ -127,258 +144,252 @@ First Time Setup:
 
 Run Build and Run for first time.bat:
 
+Code snippet
+
 cd "<project folder>"
 git clone [https://github.com/hapifhir/hapi-fhir-jpaserver-starter.git](https://github.com/hapifhir/hapi-fhir-jpaserver-starter.git) hapi-fhir-jpaserver
-copy .\hapi-fhir-Setup\target\classes\application.yaml .\hapi-fhir-jpaserver\target\classes\application.yaml
+copy .\\hapi-fhir-Setup\\target\\classes\\application.yaml .\\hapi-fhir-jpaserver\\target\\classes\\application.yaml
 mvn clean package -DskipTests=true -Pboot
 docker-compose build --no-cache
 docker-compose up -d
-
 This clones the HAPI FHIR server, copies configuration, builds the project, and starts the containers.
 
 Subsequent Runs:
 
 Run Run.bat:
 
+Code snippet
+
 cd "<project folder>"
 docker-compose up -d
-
 This starts the Flask app (port 5000) and HAPI FHIR server (port 8080).
 
 Access the Application:
 
 Flask UI: http://localhost:5000
-
 HAPI FHIR server: http://localhost:8080
-
 Manual Setup (Linux/MacOS/Windows):
 
 Preparation (Standalone Version Only):
 
+Bash
+
 cd <project folder>
 git clone [https://github.com/hapifhir/hapi-fhir-jpaserver-starter.git](https://github.com/hapifhir/hapi-fhir-jpaserver-starter.git) hapi-fhir-jpaserver
 cp ./hapi-fhir-Setup/target/classes/application.yaml ./hapi-fhir-jpaserver/target/classes/application.yaml
-
 Build:
 
+Bash
+
 # Build HAPI FHIR (Standalone Version Only)
 mvn clean package -DskipTests=true -Pboot
 
 # Build Docker Image (Specify APP_MODE=lite in docker-compose.yml for Lite version)
 docker-compose build --no-cache
-
 Run:
 
-docker-compose up -d
+Bash
 
+docker-compose up -d
 Access the Application:
 
 Flask UI: http://localhost:5000
-
 HAPI FHIR server (Standalone only): http://localhost:8080
-
 Local Development (Without Docker):
 
 Clone the Repository:
 
+Bash
+
 git clone [https://github.com/Sudo-JHare/FHIRFLARE-IG-Toolkit.git](https://github.com/Sudo-JHare/FHIRFLARE-IG-Toolkit.git)
 cd FHIRFLARE-IG-Toolkit
-
 Install Dependencies:
 
+Bash
+
 python -m venv venv
 source venv/bin/activate  # On Windows: venv\Scripts\activate
 pip install -r requirements.txt
-
 Install Node.js, GoFSH, and SUSHI (for FSH Converter):
 
+Bash
+
 # Example for Debian/Ubuntu
 curl -fsSL [https://deb.nodesource.com/setup_18.x](https://deb.nodesource.com/setup_18.x) | sudo bash -
 sudo apt-get install -y nodejs
 # Install globally
 npm install -g gofsh fsh-sushi
-
 Set Environment Variables:
 
+Bash
+
 export FLASK_SECRET_KEY='your-secure-secret-key'
 export API_KEY='your-api-key'
 # Optional: Set APP_MODE to 'lite' if desired
 # export APP_MODE='lite'
-
 Initialize Directories:
 
+Bash
+
 mkdir -p instance static/uploads logs
 # Ensure write permissions if needed
 # chmod -R 777 instance static/uploads logs
-
 Run the Application:
 
+Bash
+
 export FLASK_APP=app.py
 flask run
-
 Access at http://localhost:5000.
 
 Usage
 Import an IG
-Navigate to Import IG (/import-ig).
+### Search, View Details, and Import Packages
+Navigate to **Search and Import Packages** (`/search-and-import`).
+1.  The page will load a list of available FHIR Implementation Guide packages from a local cache or by fetching from configured registries.
+    * A loading animation and progress messages are shown if fetching from registries.
+    * The timestamp of the last cache update is displayed.
+2.  Use the search bar to filter packages by name or author.
+3.  Packages are paginated for easier Browse.
+4.  For each package, you can:
+    * View its latest official and absolute versions.
+    * Click on the package name to navigate to a **detailed view** (`/package-details/<name>`) showing:
+        * Comprehensive metadata (author, FHIR version, canonical URL, description).
+        * A full list of available versions with publication dates.
+        * Declared dependencies.
+        * Other packages that depend on it (dependents).
+        * Version history (logs).
+    * Directly import a specific version using the "Import" button on the search page or the details page.
+5.  **Cache Management:**
+    * A "Clear & Refresh Cache" button is available to trigger a background task (`/api/refresh-cache-task`) that clears the local database and in-memory cache and fetches the latest package information from all configured registries. Progress is shown via a live log.
 
 Enter a package name (e.g., hl7.fhir.au.core) and version (e.g., 1.1.0-preview).
-
 Choose a dependency mode:
-
 Current Recursive: Import all dependencies listed in package.json recursively.
-
 Patch Canonical Versions: Import only canonical FHIR packages (e.g., hl7.fhir.r4.core).
-
 Tree Shaking: Import only dependencies containing resources actually used by the main package.
-
 Click Import to download the package and dependencies.
-
 Manage IGs
 Go to Manage FHIR Packages (/view-igs) to view downloaded and processed IGs.
 
 Actions:
-
 Process: Extract metadata (resource types, profiles, must-support elements, examples).
-
 Unload: Remove processed IG data from the database.
-
 Delete: Remove package files from the filesystem.
-
 Duplicates are highlighted for resolution.
-
 View Processed IGs
 After processing, view IG details (/view-ig/<id>), including:
 
 Resource types and profiles.
-
 Must-support elements and examples.
-
 Profile relationships (compliesWithProfile, imposeProfile) if enabled (DISPLAY_PROFILE_RELATIONSHIPS).
-
 Interactive StructureDefinition viewer (Differential, Snapshot, Must Support, Key Elements, Constraints, Terminology, Search Params).
-
 Validate FHIR Resources/Bundles
 Navigate to Validate FHIR Sample (/validate-sample).
 
 Select a package (e.g., hl7.fhir.au.core#1.1.0-preview).
-
 Choose Single Resource or Bundle mode.
-
 Paste or upload FHIR JSON/XML (e.g., a Patient resource).
-
+Submit to view validation errors/warnings. Note: Alpha feature; report issues to GitHub (remove PHI).
-Submit to view validation errors/warnings.
-
-Note: Alpha feature; report issues to GitHub (remove PHI).
-
 Push IGs to a FHIR Server
 Go to Push IGs (/push-igs).
 
 Select a downloaded package.
-
 Enter the Target FHIR Server URL.
-
 Configure Authentication (None, Bearer Token).
-
 Choose options: Include Dependencies, Force Upload (skips comparison check), Dry Run, Verbose Log.
-
 Optionally filter by Resource Types (comma-separated) or Skip Specific Files (paths within package, comma/newline separated).
-
 Click Push to FHIR Server to upload resources. Canonical resources are checked before upload. Identical resources are skipped unless Force Upload is checked.
-
 Monitor progress in the live console.
-
 Upload Test Data
 Navigate to Upload Test Data (/upload-test-data).
 
 Enter the Target FHIR Server URL.
-
 Configure Authentication (None, Bearer Token).
-
 Select one or more .json, .xml files, or a single .zip file containing test resources.
-
 Optionally check Validate Resources Before Upload? and select a Validation Profile Package.
-
 Choose Upload Mode:
-
 Individual Resources: Uploads each resource one by one in dependency order.
-
 Transaction Bundle: Uploads all resources in a single transaction.
-
 Optionally check Use Conditional Upload (Individual Mode Only)? to use If-Match headers for updates.
-
 Choose Error Handling:
-
 Stop on First Error: Halts the process if any validation or upload fails.
-
 Continue on Error: Reports errors but attempts to process/upload remaining resources.
-
 Click Upload and Process. The tool parses files, optionally validates, analyzes dependencies, topologically sorts resources, and uploads them according to selected options.
-
 Monitor progress in the streaming log output.
-
 Convert FHIR to FSH
 Navigate to FSH Converter (/fsh-converter).
 
 Optionally select a package for context (e.g., hl7.fhir.au.core#1.1.0-preview).
-
 Choose input mode:
-
 Upload File: Upload a FHIR JSON/XML file.
-
 Paste Text: Paste FHIR JSON/XML content.
-
 Configure options:
-
 Output Style: file-per-definition, group-by-fsh-type, group-by-profile, single-file.
-
 Log Level: error, warn, info, debug.
-
 FHIR Version: R4, R4B, R5, or auto-detect.
-
 Fishing Trip: Enable round-trip validation with SUSHI, generating a comparison report.
-
 Dependencies: Specify additional packages (e.g., hl7.fhir.us.core@6.1.0, one per line).
-
 Indent Rules: Enable context path indentation for readable FSH.
-
 Meta Profile: Choose only-one, first, or none for meta.profile handling.
-
 Alias File: Upload an FSH file with aliases (e.g., $MyAlias = http://example.org).
-
 No Alias: Disable automatic alias generation.
-
 Click Convert to FSH to generate and display FSH output, with a waiting spinner (light/dark theme) during processing.
-
 If Fishing Trip is enabled, view the comparison report via the "Click here for SUSHI Validation" badge button.
-
 Download the result as a .fsh file.
+Retrieve and Split Bundles
+Navigate to Retrieve/Split Data (/retrieve-split-data).
 
+Retrieve Bundles from Server:
+
+Enter the FHIR Server URL (defaults to the proxy if empty).
+Select one or more Resource Types to retrieve (e.g., Patient, Observation).
+Optionally check Fetch Referenced Resources.
+If checked, further optionally check Fetch Full Reference Bundles to retrieve entire bundles for each referenced type (e.g., all Patients if a Patient is referenced) instead of individual resources by ID.
+Click Retrieve Bundles.
+Monitor progress in the streaming log. A ZIP file containing the retrieved bundles/resources will be prepared for download.
+Split Uploaded Bundles:
+
+Upload a ZIP file containing FHIR bundles (JSON format).
+Click Split Bundles.
+A ZIP file containing individual resources extracted from the bundles will be prepared for download.
 Explore FHIR Operations
 Navigate to FHIR UI Operations (/fhir-ui-operations).
 
 Toggle between local HAPI (/fhir) or a custom FHIR server.
-
 Click Fetch Metadata to load the server’s CapabilityStatement.
-
 Select a resource type (e.g., Patient, Observation) or System to view operations:
-
 System operations: GET /metadata, POST /, GET /_history, GET/POST /$diff, POST /$reindex, POST /$expunge, etc.
-
 Resource operations: GET Patient/:id, POST Observation/_search, etc.
-
 Use Try it out to input parameters or request bodies, then Execute to view results in JSON, XML, or narrative formats.
 
+### Configure Embedded HAPI FHIR Server (Standalone Mode)
+For users running the **Standalone version**, which includes an embedded HAPI FHIR server.
+1.  Navigate to **Configure HAPI FHIR** (`/config-hapi`).
+2.  The page displays the content of the HAPI FHIR server's `application.yaml` file.
+3.  You can edit the configuration directly in the text area.
+    * *Caution: Incorrect modifications can break the HAPI FHIR server.*
+4.  Click **Save Configuration** to apply your changes to the `application.yaml` file.
+5.  Click **Restart Tomcat** to restart the HAPI FHIR server and load the new configuration. The restart process may take a few moments.
+
 API Usage
 Import IG
+Bash
+
 curl -X POST http://localhost:5000/api/import-ig \
 -H "Content-Type: application/json" \
 -H "X-API-Key: your-api-key" \
 -d '{"package_name": "hl7.fhir.au.core", "version": "1.1.0-preview", "dependency_mode": "recursive"}'
-
 Returns complies_with_profiles, imposed_profiles, and duplicate_packages_present info.
 
+### Refresh Package Cache (Background Task)
+```bash
+curl -X POST http://localhost:5000/api/refresh-cache-task \
+-H "X-API-Key: your-api-key"
+
 Push IG
+Bash
+
 curl -X POST http://localhost:5000/api/push-ig \
 -H "Content-Type: application/json" \
 -H "Accept: application/x-ndjson" \
@@ -393,10 +404,11 @@ curl -X POST http://localhost:5000/api/push-ig \
       "verbose": false,
       "auth_type": "none"
     }'
-
 Returns a streaming NDJSON response with progress and final summary.
 
 Upload Test Data
+Bash
+
 curl -X POST http://localhost:5000/api/upload-test-data \
 -H "X-API-Key: your-api-key" \
 -H "Accept: application/x-ndjson" \
@@ -410,10 +422,29 @@ curl -X POST http://localhost:5000/api/upload-test-data \
 -F "use_conditional_uploads=true" \
 -F "test_data_files=@/path/to/your/patient.json" \
 -F "test_data_files=@/path/to/your/observations.zip"
+Returns a streaming NDJSON response with progress and final summary. Uses multipart/form-data for file uploads.
 
-Returns a streaming NDJSON response with progress and final summary.
+Retrieve Bundles
+Bash
 
-Uses multipart/form-data for file uploads.
+curl -X POST http://localhost:5000/api/retrieve-bundles \
+-H "X-API-Key: your-api-key" \
+-H "Accept: application/x-ndjson" \
+-F "fhir_server_url=http://your-fhir-server/fhir" \
+-F "resources=Patient" \
+-F "resources=Observation" \
+-F "validate_references=true" \
+-F "fetch_reference_bundles=false"
+Returns a streaming NDJSON response with progress. The X-Zip-Path header in the final response part will contain the path to download the ZIP archive (e.g., /tmp/retrieved_bundles_datetime.zip).
+
+Split Bundles
+Bash
+
+curl -X POST http://localhost:5000/api/split-bundles \
+-H "X-API-Key: your-api-key" \
+-H "Accept: application/x-ndjson" \
+-F "split_bundle_zip_path=@/path/to/your/bundles.zip"
+Returns a streaming NDJSON response. The X-Zip-Path header in the final response part will contain the path to download the ZIP archive of split resources.
 
 Validate Resource/Bundle
 Not yet exposed via API; use the UI at /validate-sample.
@@ -422,181 +453,132 @@ Configuration Options
 Located in app.py:
 
 VALIDATE_IMPOSED_PROFILES: (Default: True) Validates resources against imposed profiles during push.
-
 DISPLAY_PROFILE_RELATIONSHIPS: (Default: True) Shows compliesWithProfile and imposeProfile in the UI.
-
 FHIR_PACKAGES_DIR: (Default: /app/instance/fhir_packages) Stores .tgz packages and metadata.
-
 UPLOAD_FOLDER: (Default: /app/static/uploads) Stores GoFSH output files and FSH comparison reports.
-
 SECRET_KEY: Required for CSRF protection and sessions. Set via environment variable or directly.
-
 API_KEY: Required for API authentication. Set via environment variable or directly.
-
 MAX_CONTENT_LENGTH: (Default: Flask default) Max size for HTTP request body (e.g., 16 * 1024 * 1024 for 16MB). Important for large uploads.
-
 MAX_FORM_PARTS: (Default: Werkzeug default, often 1000) Default max number of form parts. Overridden for /api/upload-test-data by CustomFormDataParser.
 
+### Get HAPI FHIR Configuration (Standalone Mode)
+```bash
+curl -X GET http://localhost:5000/api/config \
+-H "X-API-Key: your-api-key"
+
+Save HAPI FHIR Configuration:
+curl -X POST http://localhost:5000/api/config \
+-H "Content-Type: application/json" \
+-H "X-API-Key: your-api-key" \
+-d '{"your_yaml_key": "your_value", ...}' # Send the full YAML content as JSON
+
+Restart HAPI FHIR Server:
+curl -X POST http://localhost:5000/api/restart-tomcat \
+-H "X-API-Key: your-api-key"
+
 Testing
 The project includes a test suite covering UI, API, database, file operations, and security.
 
 Test Prerequisites:
 
 pytest: For running tests.
-
+pytest-mock: For mocking dependencies. Install: pip install pytest pytest-mock
-pytest-mock: For mocking dependencies.
-
-Install: pip install pytest pytest-mock
-
 Running Tests:
 
+Bash
+
 cd <project folder>
 pytest tests/test_app.py -v
-
 Test Coverage:
 
-UI Pages: Homepage, Import IG, Manage IGs, Push IGs, Validate Sample, View Processed IG, FSH Converter, Upload Test Data.
+UI Pages: Homepage, Import IG, Manage IGs, Push IGs, Validate Sample, View Processed IG, FSH Converter, Upload Test Data, Retrieve/Split Data.
-
+API Endpoints: POST /api/import-ig, POST /api/push-ig, GET /get-structure, GET /get-example, POST /api/upload-test-data, POST /api/retrieve-bundles, POST /api/split-bundles.
-API Endpoints: POST /api/import-ig, POST /api/push-ig, GET /get-structure, GET /get-example, POST /api/upload-test-data.
-
 Database: IG processing, unloading, viewing.
-
+File Operations: Package processing, deletion, FSH output, ZIP handling.
-File Operations: Package processing, deletion, FSH output.
-
 Security: CSRF protection, flash messages, secret key.
-
 FSH Converter: Form submission, file/text input, GoFSH execution, Fishing Trip comparison.
-
 Upload Test Data: Parsing, dependency graph, sorting, upload modes, validation, conditional uploads.
-
 Development Notes
 Background
-The toolkit addresses the need for a comprehensive FHIR IG management tool, with recent enhancements for resource validation, FSH conversion with advanced GoFSH features, flexible versioning, improved IG pushing, and dependency-aware test data uploading, making it a versatile platform for FHIR developers.
+The toolkit addresses the need for a comprehensive FHIR IG management tool, with recent enhancements for resource validation, FSH conversion with advanced GoFSH features, flexible versioning, improved IG pushing, dependency-aware test data uploading, and bundle retrieval/splitting, making it a versatile platform for FHIR developers.
 
 Technical Decisions
 Flask: Lightweight and flexible for web development.
-
 SQLite: Simple for development; consider PostgreSQL for production.
-
 Bootstrap 5.3.3: Responsive UI with custom styling.
-
 Lottie-Web: Renders themed animations for FSH conversion waiting spinner.
-
 GoFSH/SUSHI: Integrated via Node.js for advanced FSH conversion and round-trip validation.
-
 Docker: Ensures consistent deployment with Flask and HAPI FHIR.
-
 Flexible Versioning: Supports non-standard IG versions (e.g., -preview, -ballot).
-
+Live Console/Streaming: Real-time feedback for complex operations (Push, Upload Test Data, FSH, Retrieve Bundles).
-Live Console/Streaming: Real-time feedback for complex operations (Push, Upload Test Data, FSH).
-
 Validation: Alpha feature with ongoing FHIRPath improvements.
-
 Dependency Management: Uses topological sort for Upload Test Data feature.
-
 Form Parsing: Uses custom Werkzeug parser for Upload Test Data to handle large numbers of files.
-
 Recent Updates
+* Enhanced package search page with caching, detailed views (dependencies, dependents, version history), and background cache refresh.
 Upload Test Data Enhancements (April 2025):
-
 Added optional Pre-Upload Validation against selected IG profiles.
-
 Added optional Conditional Uploads (GET + POST/PUT w/ If-Match) for individual mode.
-
 Implemented robust XML parsing using fhir.resources library (when available).
-
 Fixed 413 Request Entity Too Large errors for large file counts using a custom Werkzeug FormDataParser.
-
 Path: templates/upload_test_data.html, app.py, services.py, forms.py.
-
 Push IG Enhancements (April 2025):
-
 Added semantic comparison to skip uploading identical resources.
-
 Added "Force Upload" option to bypass comparison.
-
 Improved handling of canonical resources (search before PUT/POST).
-
 Added filtering by specific files to skip during push.
-
 More detailed summary report in stream response.
-
 Path: templates/cp_push_igs.html, app.py, services.py.
-
 Waiting Spinner for FSH Converter (April 2025):
-
 Added a themed (light/dark) Lottie animation spinner during FSH execution.
-
 Path: templates/fsh_converter.html, static/animations/, static/js/lottie-web.min.js.
-
 Advanced FSH Converter (April 2025):
-
 Added support for GoFSH advanced options: --fshing-trip, --dependency, --indent, --meta-profile, --alias-file, --no-alias.
-
 Displays Fishing Trip comparison reports.
-
 Path: templates/fsh_converter.html, app.py, services.py, forms.py.
-
+(New) Retrieve and Split Data (May 2025):
-(Keep older updates listed here if desired)
+Added UI and API for retrieving bundles from a FHIR server by resource type.
-
+Added options to fetch referenced resources (individually or as full type bundles).
+Added functionality to split uploaded ZIP files of bundles into individual resources.
+Streaming log for retrieval and ZIP download for results.
+Paths: templates/retrieve_split_data.html, app.py, services.py, forms.py.
 Known Issues and Workarounds
 Favicon 404: Clear browser cache or verify /app/static/favicon.ico.
-
 CSRF Errors: Set FLASK_SECRET_KEY and ensure {{ form.hidden_tag() }} in forms.
-
 Import Fails: Check package name/version and connectivity.
-
 Validation Accuracy: Alpha feature; report issues to GitHub (remove PHI).
-
 Package Parsing: Non-standard .tgz filenames may parse incorrectly. Fallback uses name-only parsing.
-
 Permissions: Ensure instance/ and static/uploads/ are writable.
-
 GoFSH/SUSHI Errors: Check ./logs/flask_err.log for ERROR:services:GoFSH failed. Ensure valid FHIR inputs and SUSHI installation.
-
 Upload Test Data XML Parsing: Relies on fhir.resources library for full validation; basic parsing used as fallback. Complex XML structures might not be fully analyzed for dependencies with basic parsing. Prefer JSON for reliable dependency analysis.
-
 413 Request Entity Too Large: Primarily handled by CustomFormDataParser for /api/upload-test-data. Check the parser's max_form_parts limit if still occurring. MAX_CONTENT_LENGTH in app.py controls overall size. Reverse proxy limits (client_max_body_size in Nginx) might also apply.
 
+
 Future Improvements
 Upload Test Data: Improve XML parsing further (direct XML->fhir.resource object if possible), add visual progress bar, add upload order preview, implement transaction bundle size splitting, add 'Clear Target Server' option (with confirmation).
-
 Validation: Enhance FHIRPath for complex constraints; add API endpoint.
-
 Sorting: Sort IG versions in /view-igs (e.g., ascending).
-
 Duplicate Resolution: Options to keep latest version or merge resources.
-
 Production Database: Support PostgreSQL.
-
 Error Reporting: Detailed validation error paths in the UI.
-
 FSH Enhancements: Add API endpoint for FSH conversion; support inline instance construction.
-
 FHIR Operations: Add complex parameter support (e.g., /$diff with left/right).
-
+Retrieve/Split Data: Add option to filter resources during retrieval (e.g., by date, specific IDs).
 Completed Items
 Testing suite with basic coverage.
-
 API endpoints for POST /api/import-ig and POST /api/push-ig.
-
 Flexible versioning (-preview, -ballot).
-
 CSRF fixes for forms.
-
 Resource validation UI (alpha).
-
 FSH Converter with advanced GoFSH features and waiting spinner.
-
 Push IG enhancements (force upload, semantic comparison, canonical handling, skip files).
-
 Upload Test Data feature with dependency sorting, multiple upload modes, pre-upload validation, conditional uploads, robust XML parsing, and fix for large file counts.
-
+Retrieve and Split Data functionality with reference fetching and ZIP download.
 Far-Distant Improvements
 Cache Service: Use Redis for IG metadata caching.
-
 Database Optimization: Composite index on ProcessedIg.package_name and ProcessedIg.version.
 
+
 Directory Structure
 FHIRFLARE-IG-Toolkit/
 ├── app.py                              # Main Flask application
@@ -608,7 +590,7 @@ FHIRFLARE-IG-Toolkit/
 ├── README.md                           # Project documentation
 ├── requirements.txt                    # Python dependencies
 ├── Run.bat                             # Windows script for running Docker
-├── services.py                         # Logic for IG import, processing, validation, pushing, FSH conversion, test data upload
+├── services.py                         # Logic for IG import, processing, validation, pushing, FSH conversion, test data upload, retrieve/split
 ├── supervisord.conf                    # Supervisor configuration
 ├── hapi-fhir-Setup/
 │   ├── README.md                       # HAPI FHIR setup instructions
@@ -649,38 +631,31 @@ FHIRFLARE-IG-Toolkit/
 │   ├── fsh_converter.html              # UI for FSH conversion
 │   ├── import_ig.html                  # UI for importing IGs
 │   ├── index.html                      # Homepage
+│   ├── retrieve_split_data.html        # UI for Retrieve and Split Data
 │   ├── upload_test_data.html           # UI for Uploading Test Data
 │   ├── validate_sample.html            # UI for validating resources/bundles
+│   ├── config_hapi.html                # UI for HAPI FHIR Configuration
 │   └── _form_helpers.html              # Form helper macros
 ├── tests/
 │   └── test_app.py                     # Test suite
 └── hapi-fhir-jpaserver/                # HAPI FHIR server resources (if Standalone)
 
+Contributing
+Fork the repository.
+Create a feature branch (git checkout -b feature/your-feature).
+Commit changes (git commit -m "Add your feature").
+Push to your branch (git push origin feature/your-feature).
+Open a Pull Request.
+Ensure code follows PEP 8 and includes tests in tests/test_app.py.
 
-## Contributing
+Troubleshooting
-
+Favicon 404: Clear browser cache or verify /app/static/favicon.ico: docker exec -it <container_name> curl http://localhost:5000/static/favicon.ico
-* Fork the repository.
+CSRF Errors: Set FLASK_SECRET_KEY and ensure {{ form.hidden_tag() }} in forms.
-* Create a feature branch (`git checkout -b feature/your-feature`).
+Import Fails: Check package name/version and connectivity.
-* Commit changes (`git commit -m "Add your feature"`).
+Validation Accuracy: Alpha feature; report issues to GitHub (remove PHI).
-* Push to your branch (`git push origin feature/your-feature`).
+Package Parsing: Non-standard .tgz filenames may parse incorrectly. Fallback uses name-only parsing.
-* Open a Pull Request.
+Permissions: Ensure instance/ and static/uploads/ are writable: chmod -R 777 instance static/uploads logs
-
+GoFSH/SUSHI Errors: Check ./logs/flask_err.log for ERROR:services:GoFSH failed. Ensure valid FHIR inputs and SUSHI installation: docker exec -it <container_name> sushi --version
-Ensure code follows PEP 8 and includes tests in `tests/test_app.py`.
+413 Request Entity Too Large: Increase MAX_CONTENT_LENGTH and MAX_FORM_PARTS in app.py. If using a reverse proxy (e.g., Nginx), increase its client_max_body_size setting as well. Ensure the application/container is fully restarted/rebuilt.
-
+License
-## Troubleshooting
+Licensed under the Apache 2.0 License. See LICENSE.md for details.
-
-* **Favicon 404:** Clear browser cache or verify `/app/static/favicon.ico`:
-    `docker exec -it <container_name> curl http://localhost:5000/static/favicon.ico`
-* **CSRF Errors:** Set `FLASK_SECRET_KEY` and ensure `{{ form.hidden_tag() }}` in forms.
-* **Import Fails:** Check package name/version and connectivity.
-* **Validation Accuracy:** Alpha feature; report issues to GitHub (remove PHI).
-* **Package Parsing:** Non-standard `.tgz` filenames may parse incorrectly. Fallback uses name-only parsing.
-* **Permissions:** Ensure `instance/` and `static/uploads/` are writable:
-    `chmod -R 777 instance static/uploads logs`
-* **GoFSH/SUSHI Errors:** Check `./logs/flask_err.log` for `ERROR:services:GoFSH failed`. Ensure valid FHIR inputs and SUSHI installation:
-    `docker exec -it <container_name> sushi --version`
-* **413 Request Entity Too Large:** Increase `MAX_CONTENT_LENGTH` and `MAX_FORM_PARTS` in `app.py`. If using a reverse proxy (e.g., Nginx), increase its `client_max_body_size` setting as well. Ensure the application/container is fully restarted/rebuilt.
-
-## License
-
-Licensed under the Apache 2.0 License. See `LICENSE.md` for details.