Deployment Change Log c41ab62a8

The separate integrity check periodic task has been removed. Its functionality is now integrated into the main Zoho Analytics refresh task by passing `integrity_check_first=True` in its task arguments.

This commit introduces a new feature allowing users to run an integrity check on Zoho Analytics data before a full refresh. It also refactors the integrity check process for better control and separation of concerns.

**Key Changes:**

*   **Pre-Refresh Integrity Check (dear_zoho_analytics):**
    *   Added `integrity_check_first` option to the `RefreshZohoAnalyticsForm` and the `refresh_dear_analytics_zoho` and `refresh_zoho_analytics` tasks.
    *   When enabled, the `zoho_analytics_integrity_check` task is now called for each instance *before* the main data refresh, allowing for early detection and potential self-healing of data inconsistencies.
*   **Integrity Check Refactoring (dear_zoho_analytics):**
    *   **Decoupled Refresh Triggers:** Individual integrity check functions (e.g., `fix_missing_sales_documents`, `find_lastmod_mismatch_customers`) no longer directly trigger Zoho Analytics refreshes. Instead, they now return an optional `datetime` indicating the earliest point from which a refresh is suggested if inconsistencies are found. This allows the calling task (`zoho_analytics_integrity_check` or `refresh_dear_analytics_zoho`) to orchestrate the refresh based on the accumulated suggestions.
    *   **Unified Start Date Suggestion:** The `find_missing_or_mismatched_documents` task now aggregates suggested refresh dates from various sub-checks and returns the earliest one.
    *   **Simplified Per-Instance Processing:** The `find_missing_or_mismatched_documents` task was modified to process a single Dear instance, aligning with how the main `zoho_analytics_integrity_check` task iterates through selected instances.
*   **New Enum for Integrity Modes (dear_zoho_analytics):**
    *   Introduced `IntegrityMode` enum (`FULL`, `EXCLUDE_PRODUCT_MOVEMENTS`) for clearer definition of integrity check scopes.
*   **Cache Refresh Logic Adjustment (dear_zoho_analytics):**
    *   Ensured "products" and "customers" are always included in Dear cache refresh topics during a Zoho Analytics refresh.

This refactoring enhances the robustness and maintainability of the Zoho Analytics integration by providing a more controlled and explicit integrity verification step.

This commit refactors how Redis locks are used in the `dear_zoho_analytics` app:

*   **Instance-specific Locks:** Redis lock keys for various analytics refresh tasks (e.g., suppliers, customers, products, sales, purchases, product movements, stock transfers, stock adjustments, product availability, product families, customer tags, product tags) are now made instance-specific by including the `dear_instance_name` in the lock key. This prevents jobs for different Dear instances from blocking each other.
*   **Simplified Lock Parameters:** Explicit `timeout`, `expire`, and `auto_renewal` parameters have been removed from most `acquire_redis_lock_with_timeout` calls, relying on the function's default values for cleaner code.
*   **Duplicate Product Movement Check Lock:** The Redis lock for `zoho_analytics_delete_duplicate_product_movement_rows` has been moved from `tasks.py` into the `zoho_integrity_verification.py` module, applying the lock per Dear instance for concurrent processing.
*   **Sales Mismatch Lock Key:** The lock key for `ANALYTICS-UPDATE-SALES-FIND_MISMATCH` has also been updated to be instance-specific.
*   **Corrected Product Movement Lock Key:** The lock key for `ANALYTICS-UPDATE-PRODUCT-MOVEMENTS` in `zoho_backend_dear.py` now correctly uses `dear_cached_api.entity` (instance name) instead of `dear_cached_api.account_id`.

# Conflicts:
#	ansible/includes/version.yml
#	core/common_files/api_helpers.py
#	project/celery.py

**core/**:
- Modified `_get_default_lock_metadata` to include the hostname in the default lock metadata and allow for enriching existing metadata dictionaries with `job_id` (if in a Celery task) and `hostname`.
- Updated `acquire_redis_lock_with_timeout` to utilize this enhanced metadata logic, ensuring `hostname` is always part of the lock metadata where applicable.

**project/**:
- Corrected the method for retrieving Celery worker queue names in `worker_ready_probe` to use `sender.app.conf.get('task_queues')`, which is more robust and compatible with modern Celery versions.

- **cached_dear/tilecloud_shopify_interentity.py**: Added logging messages to `_verify_xero_entity` to indicate the start and end of fetching Xero accounts and the number of active accounts found. This improves visibility during the Xero integration verification process.
- **core/common_files/api_helpers.py**: Added informative logging within the `is_gevent_monkey_patched` function to indicate which detection method successfully identified gevent monkey-patching. This aids debugging and understanding of the gevent environment.

*   **Redis Locks (`acquire_redis_lock_with_timeout`):**
    *   Added a diagnostic check to warn and disable `auto_renewal` when running in a Gevent worker, as auto-renewal is unreliable in that environment. The expiry is set to 4 hours in this scenario.
    *   Updated type hint for `lock_metadata`.
*   **Task Decorator (`no_simultaneous_execution_no_retry`):**
    *   Increased the Redis lock `TIMEOUT` from 120 seconds to 4 hours for long-running tasks.
    *   Explicitly disabled `auto_renewal` for the lock in this decorator to ensure the lock persists for the entire task duration, addressing Gevent reliability concerns.
    *   Fixed a minor typo in the docstring.

# Conflicts:
#	ansible/includes/version.yml
#	ansible/templates/supervisor_app.conf.jinja2

This commit introduces the following changes:

**`cached_dear` app:**
*   Increased `max_retries` for Xero API calls from 3 to 5.
*   Improved Xero API rate limit logging to include `X-AppMinLimit-Remaining` header.
*   Added specific error handling for Xero daily rate limit exhaustion (unrecoverable `X-DayLimit-Remaining` being 0).
*   Refined the log message for 429 (rate limit) errors to be more informative.

**`dear_zoho_analytics` app:**
*   Removed commented-out and problematic `sales_lines_lifecycle` integrity check code.
*   Refactored the `find_mismatch_sales_order_lines` logging to:
    *   Collect all problematic `order_guid`s from Zoho in one pass.
    *   Fetch corresponding order numbers (`string_uniqueID`) from `DearCache` in a single query for better performance.
    *   Enhance log messages to include the order number and the Dear instance name for better context and readability.

This commit introduces several improvements primarily to the `dear_zoho_analytics` app, with a minor update in `cached_dear`.

**dear_zoho_analytics:**
-   **Integrity Check Enhancements:**
    -   Implemented a new 'Sales Only' mode for the Zoho Analytics integrity check, allowing users to run a faster check that skips product movement data verification.
    -   Refactored the duplicate and invalid data deletion functions (`zoho_analytics_fix_missing_and_duplicate_master_data`, `zoho_analytics_delete_duplicate_sales_fact_rows`, etc.) to accept `selected_dear_account_ids`. This enables processing specific Dear instances, making the tasks more efficient and removing redundant outer loops.
    -   Improved logging within integrity checks, adding Dear instance names to messages and providing more specific details (e.g., actual sale numbers for missing orders) for better traceability.
    -   Enabled immediate Zoho sales data refresh for missing sales documents detected during integrity checks, ensuring prompt data consistency.
    -   Updated the `run_integrity_check_form.html` template to include dedicated buttons for 'Run Full Check' and 'Run Sales Check Only'.
    -   Ensured the `refresh_zoho_analytics` task correctly passes the new `verbose` flag and properly triggers product availability regeneration for relevant instances.
-   **Verbose Logging:**
    -   Introduced a `verbose` parameter to `refresh_dear_analytics_zoho` and `refresh_sales_tables` functions and their callers, allowing for more detailed progress logging during Zoho data refreshes, especially for sales processing.

**cached_dear:**
-   **Error Handling Clarity:**
    -   Updated the docstring and comments for `handle_xero_api_errors` in `xero_journal_helpers.py` to explicitly mention limiting retries and logging 429 errors with 'Retry-After' header content.

**project:**
-   **Development/Testing Utility:**
    -   Added logic to `local_settings.py` to conditionally load production database credentials from a YAML file. This feature allows developers to connect to production-like databases for testing and debugging purposes by setting specific environment variables (`SHORT_HOSTNAME` and `USE_PROD_DB`).

Introduces the ability to run Zoho Analytics integrity checks on specific Cin7 Core (Dear) instances, rather than always on all configured instances.

**Changes:**
- **`cached_dear/`**: Modified `get_dear_settings_from_instance_set` to allow filtering by a list of `selected_instances`.
- **`dear_zoho_analytics/`**:
    - Updated various integrity check Celery tasks (`zoho_analytics_integrity_check`, `task_zoho_analytics_delete_duplicate_master_data`, `find_missing_or_mismatched_documents`, `zoho_analytics_integrity_check_product_movements`) to accept an optional `selected_dear_account_ids` argument.
    - These tasks now use the enhanced `get_dear_settings_from_instance_set` function to process only the selected instances.
    - Added error handling within tasks to gracefully skip instances where the Zoho client cannot be initialized.
    - Refactored the `zoho_analytics_integrity_check` view to display a form for selecting instances before running the check.
    - Added a new template `run_integrity_check_form.html` for the instance selection form.
    - Updated `FormView` form class references to use an aliased import for `forms` to prevent conflicts.

**App:** `cached_dear`

This change generalizes the intercompany journal synchronization process within the `cached_dear` app.

Key changes:
- The `task_run_intercompany_journal_sync_pipeline` now accepts a `sync_xero_instance_pairs` argument, allowing the task to be configured for multiple parent-child Xero instance pairs, rather than being hardcoded to "tilecloud" and "yabby".
- Periodic task setup (`setup_periodic_tasks`) is updated to pass the `sync_xero_instance_pairs` list to the pipeline task.
- Minor docstring and comment cleanup for clarity.

**cached_dear:**
*   Enhanced Dear Systems API pagination logic to ensure `total_records` is correctly updated, even when some endpoints return an inaccurate total, by taking the maximum of `total_records` and the current page's data length.

**core:**
*   Refactored `get_celery_queue_depths` function to accept a list of queues to monitor, improving its reusability and flexibility.
*   Updated `monitor_celery_queues` task to allow specifying which queues to monitor, providing a default list of queues (based on hostname and gevent queue) if none are provided. This enhances the configurability of Celery queue monitoring.

This commit introduces a new periodic task to monitor the depth of Celery queues.

**Key changes:**
*   A new Celery task `monitor_celery_queues` is added, designed to run every minute.
*   This task retrieves the current depth of configured Celery queues from Valkey (Redis).
*   The queue depths are logged using `JobMaster.log_message`, providing metrics for monitoring systems.
*   The `setup_periodic_tasks` function is updated to schedule the new monitoring task.
*   Minor import adjustment in `core/views.py`.

Corrected a typo in the main navigation menu, changing "Force Sync C7C Void Invoices" to "Force Sync C7C Voided Invoices".

The `VoidXeroInvoicesView` class has been refactored and moved from `cached_dear/views.py` to `cached_dear/views_xero.py`. This change includes updating the URL configuration (`urls_integrations.py`) to point to the new location and adjusting imports accordingly. Additionally, the menu item text in `index.html` was updated from "Void Paid Xero Invoices" to "Force Sync C7C Void Invoices" for clarity.

The `task_void_xero_invoices` function has been updated to use the `@no_simultaneous_execution_no_retry` decorator for managing concurrent execution, replacing the manual `redis_lock` usage. This includes removing the `redis_lock` import, changing the exception handler from `redis_lock.NotAcquired` to `LockNotAcquiredAfterTimeout`, and updating the job status for lock acquisition failure from `WAIT` to `LOCK_NOT_ACQUIRED`. The task is now also configured with `bind=True` and assigned to a specific Celery queue.

Reduced CPU limits from 1000m to 500m, memory limits from 900Mi to 700Mi, and a CPU request from 500m to 250m for Valkey containers in the `valkey_pods_deployment_v2.yaml` manifest. This optimizes resource allocation for Valkey pods running on EKS.

No Django app names were identified, as the changes are in a Kubernetes deployment configuration file, not a Django application file.

This commit enhances the `LockNotAcquiredAfterTimeout` exception and its handling to provide more detailed information about the lock holder.

Key changes include:
*   The `LockNotAcquiredAfterTimeout` exception in `core/common_files/api_helpers.py` now accepts and stores `metadata` about the process holding the lock.
*   When a lock cannot be acquired, `acquire_redis_lock_with_timeout` retrieves and includes this `offending_metadata` in the exception message and logs, making it easier to diagnose which process is holding the resource.
*   Comments related to accessing exception metadata were removed from `core/celery_signals.py` as the direct enhancement to the exception object makes the metadata accessible.

Enhanced the promise date refresh workflow with Redis-based rate-limiting and job status verification to prevent duplicate or excessive task triggers. Updated task and cache handling for improved reliability.

# Conflicts:
#	ansible/includes/version.yml

(cherry picked from commit 410d09df7733ae4fbf81f3eff88366daee509ddc)

Refactored `verify_product_movements` to use a more efficient windowed and chunk-based comparison between DearCache and the Postgres analytics backend for product movements. This new approach processes products in chunks and verifies movements within specific date windows, leading to more targeted corrections (deleting invalid records or triggering fast, partial refreshes for specific product IDs and date ranges) and improved memory efficiency.

The `verify_sales_orders` function was also updated to populate `dear_order_ids` using an iterator, reducing memory consumption.

This commit introduces the following changes within the `dear_zoho_analytics` Django app:

*   **Temporarily disabled deletion of unknown records:** In both PostgreSQL and Zoho Analytics backend integrity verification processes, the automatic deletion of records found in the analytics backend but not in the Dear cache (e.g., `postgresql_records_unknown_in_dear`, `zoho_records_unknown_in_dear`) has been commented out. This is a temporary measure to address potential race conditions, and warnings are now logged.
*   **Improved memory efficiency for `DearCache` queries:** Added `.iterator()` to `DearCache.objects.all().filter(...)` queries for `PRODUCT_MOVEMENTS` in `postgres_backend.py` and `zoho_backend_dear.py` to prevent loading all objects into memory at once, improving performance for large datasets.
*   **Broadened DearCache query for Zoho integrity checks:** Removed the `filter(last_modified__gte=analytics_minimum_datetime)` from the `DearCache` query in `zoho_integrity_verification.py`. This ensures that all records in the Dear cache, not just recently modified ones, are considered when checking for missing or unwanted records in Zoho.
*   **Minor IDE configuration adjustments:** Reordered environment variables (`SHORT_HOSTNAME`, `USE_PROD_DB`) in PyCharm run configuration templates (`.idea/runConfigurations/`) for consistency.

This commit significantly optimizes the push synchronization of Dear customers to Zoho CRM accounts and contacts.

Key changes include:
*   **Batch Zoho Account Lookups:** Replaced inefficient single-record Zoho API calls for account lookup with bulk queries using Zoho's 'in' operator based on Dear GUIDs and Customer Names. This drastically reduces the number of API calls.
*   **Pre-fetching:** Zoho accounts are now pre-fetched into memory before iterating through Dear customers, allowing for fast in-memory lookups instead of repeated API calls within the processing loop.
*   **Removed Redundant Caching/Functions:** The `fetch_zoho_accounts` cache and associated single-account lookup functions (`get_zoho_account_by_name`, `get_zoho_account_by_DEAR_GUID`) have been removed.
*   **Improved Error Handling & Logging:** Enhanced logging messages and error handling, especially for `InvalidSalesRep` and general exceptions during the sync process.
*   **Refactored Processing Loop:** The main processing loop now iterates over pre-fetched Dear customer data, improving efficiency and ensuring `KeyValueJson` timestamps are updated more reliably.
*   **Batch Zoho Account Updates:** Implemented batch updates for existing Zoho accounts, sending updates in groups of 5 or more, further reducing API overhead.
*   **Safer Dictionary Access:** Used `.get()` for dictionary access to prevent `KeyError` where attributes might be missing.
*   **Corrected Tag Handling:** Improved logic for syncing customer tags to Zoho.

Reduced memory limit from 4Gi to 3Gi in `amazon_eks/base/django-api-sync-base.yaml` to optimize resource usage. Analysis of peak usage shows that 3GB will accommodate most namespaces
remove this setting from the overlay config when it was simply repeating the previous default of 4gbi

Reduced memory limit from 4Gi to 3Gi in `amazon_eks/base/django-api-sync-base.yaml` to optimize resource usage. Analysis of peak usage shows that 3GB will accommodate most namespaces

This commit introduces a robust rolling update strategy for `django-api-sync` and `worker-django-api-sync` Kubernetes deployments.

**Changes include:**
*   **`strategy-patch.yaml`**: A new patch file defining a `RollingUpdate` strategy with `maxSurge: 1` and `maxUnavailable: 0` to ensure smooth, zero-downtime deployments.
*   **`rollout-strategy-update.sh`**: A new script to automatically find and apply the `strategy-patch.yaml` to all `django-api-sync` and `worker-django-api-sync` deployments across all namespaces. It includes a `--dry-run` option for testing.
*   **`amazon_eks/base/django-api-sync-base.yaml`**: Minor YAML reordering of the `replicas` field and removal of outdated comments for cleanup.

Updated the `aider_linter.sh` script with comprehensive comments explaining its usage for pre-commit hooks, auto-testing, and ad-hoc checks. The script also now supports an `--ad-hoc` flag to stage changes before linting and references `aider_prompt.md` as the conventions file.

In `dear_zoho_analytics/`, the analytics table documentation HTML was updated to provide more specific and descriptive titles and introductory text, clarifying its purpose for Cin7 Core GrowthPath Analytics and LLM context.

**Affected Django App(s):** `dear_zoho_analytics/`

Updated the periodic task schedules for Zoho Analytics synchronization:
- Increased the main Zoho Analytics update period from 2 to 6 hours.
- Increased the integrity scan update period from 6 to 12 hours.
- The dedicated 'Refresh Cin7 Core Zoho Analytics Sales Tables' task will no longer be scheduled by this function, as its activation condition was changed to require a main update period of >= 12 hours (currently set to 6 hours).

This commit introduces the following changes:

*   **`dear_zoho_analytics`**:
    *   Renamed the analytics table documentation page title to include "and LLM Prompts."
    *   Added a note indicating the presence of LLM query prompts at the end of the documentation.
    *   Updated the description for the "Basic sales report query" LLM prompt, clarifying its purpose for Cin7 Core (DEAR Systems) and the importance of outer joins.
    *   Added a new "Schema and LLM support" dropdown menu to the main navigation, providing direct links to the "Analytics Schema Docs and LLM prompts" and "Analytics Schema Diagram" pages.
*   **Project Configuration**:
    *   Modified PyCharm run configurations (`.idea/runConfigurations/_template__of_Python.xml` and `_template__of_Python2.xml`) to adjust the presence of `SHORT_HOSTNAME` and `USE_PROD_DB` environment variables.

Adds a new `aider_linter.sh` script to lint staged files against coding standards.

Enhances the `dear_zoho_analytics/analytics_table_documentation.html` template by:
- Clarifying that the documentation can be used as context for LLMs to generate SQL queries.
- Adding an "LLM Prompts" section with an example SQL query for a basic sales report, demonstrating how to use the documented analytics tables.

**Changes:**
1.  **Kubernetes Health Check Probes:** Modified `amazon_eks/base/django-api-sync-base.yaml` to update liveness and readiness probes. They now consistently target the `/healthz/` endpoint, and the comment about the "core app" was generalized.
2.  **Valkey Pool Assignment Logic:** Updated `amazon_eks/helpers/configure_new_version.sh` to refine how `valkey_pool_number` is calculated. The previous specific remapping for pool 2 was replaced with a generalized formula (`calculated_pool_number + 5`) to map values 0-3 to 5-8.

**Django App Name(s) Identified:**
*   `core/` (Inferred from the original comment in `amazon_eks/base/django-api-sync-base.yaml` which stated the `/healthz/` endpoint was "in the core app").

- **EKS/Kubernetes:**
    - Introduced a new Kubernetes deployment (`valkey_pods_deployment_v3.yaml`) for Valkey (a Redis fork) with Sentinel for high availability, including configurations for pools 5, 6, 7, and 8.
    - Reduced CPU and memory limits for `django-api-sync` and `celery-worker` containers in `django-api-sync-base.yaml`.
    - Disabled Alertmanager in `kube-prometheus-stack-values.yaml`.
- **Django Configuration & Core Logic:**
    - Expanded the Valkey pool number range in `ansible/templates/local_settings.py.jinja2` to include Sentinel-based pools (4-8) and removed various unused imports.
    - Removed unused imports in `core/valkey_client.py`.
- **Django App `dear_zoho_analytics`:**
    - Updated the `table_description` for the `Sales_fact` model in `dear_zoho_analytics/models.py` for improved clarity regarding COGS sourcing and consistency with Cin7 Core reports.
- **IDE Settings:**
    - Adjusted environment variables in Python run configuration templates (`.idea/runConfigurations`).

Deleted the old `karpenter/provisioner.yaml` file.
Modified `karpenter/nodepool.yaml` to relax the instance generation requirement (from greater than 5 to greater than 4) and to specify `xlarge` instance sizes.

No Django app names were identified from the paths of the modified files.

This commit introduces significant improvements to Celery worker logging and configuration:

*   **JSON Logging for Celery Workers**: Configures Celery to output logs in JSON format for better machine readability and ingestion by log aggregators.
    *   Adds a new `json` formatter to `LOGGING['formatters']` in `project/settings.py`.
    *   In `project/celery.py`, the `setup_loggers` signal handler is enhanced to replace the default console logger's formatter with the new JSON formatter.
*   **Celery Queue Name Injection**: Implements a custom `CeleryQueueFilter` to automatically inject the specific queue name(s) a worker is consuming into log records.
    *   The filter is applied to all root logger handlers during the `worker_ready` signal, ensuring `%(queue)s` is available in log messages.
*   **Celery Eager Mode Disabled**: Sets `CELERY_TASK_ALWAYS_EAGER` to `False` in `project/local_settings.py`, allowing Celery tasks to be dispatched to a real worker by default in local development, rather than executed synchronously.
*   **Dependency Update**: Adds `python-json-logger` to `requirements.in`.

This commit introduces new configuration files for Karpenter, an open-source high-performance Kubernetes cluster autoscaler, designed for Amazon EKS. The changes include:
- AWS IAM policies (`controller-policy.json`, `controller-trust-policy.json`, `node-trust-policy.json`) to grant necessary permissions to the Karpenter controller and nodes.
- Karpenter Kubernetes Custom Resources (`ec2nodeclass.yaml`, `nodepool.yaml`, `provisioner.yaml`) defining how nodes should be provisioned, including AMI family, IAM roles, subnet/security group selectors, instance requirements, and scaling policies.
- A shell script (`karpenter_variables.sh`) to set essential environment variables for the Karpenter setup.
- A sample Kubernetes deployment (`test_deployment.yaml`) for testing autoscaling behavior.

These configurations are specifically tailored for the EKS cluster supporting the `django-api-sync` application.

Updates memory limits and requests for the `django-api-sync` main container and a worker container in EKS base configurations.
Removes a temporary patch in the TUI deployer that explicitly deleted `django-api-sync` related deployments before a new rollout, relying on standard K8s rollout mechanisms.

- Rename `deploy_all.sh` to `deploy.sh` and update related references.
- Move outdated deploy scripts to `old_redundant_scripts`.
- Update environment variables in IntelliJ run configuration templates.

Adds the `--force` flag to the `init_periodic_tasks` management command within the Supervisor configuration. This ensures that periodic tasks are always re-initialized during deployment, preventing potential conflicts or ensuring consistency.
```

**Django App Name(s) based on paths:** Based on the provided file paths, no explicit Django app directories (e.g., 'revel_pos/', 'core/') are present. However, the configurations are for a service identified as `django_api_sync`.

```
feat(django_api_sync): Refactor Kustomize labels and configure Valkey pool

Refactored Kustomize configurations across all overlays, moving the 'growthpath.com/valkey-pool' label from 'commonLabels' to 'generatorOptions.labels'. This standardizes label application for the deployed services. Additionally, the 'valkey_pool_number' was explicitly set to '3' for the 'test-dear' environment.
```

amazon_eks:
- Implemented persistence for average deployment step duration.
- The TUI deployer now loads the previously calculated average step time from a cache file.
- This average is used to provide more accurate Estimated Time to Completion (ETC) at the start of a deployment and during the initial steps.
- The average is updated and saved back to the cache at the end of a successful deployment.

cached_dear:
- Modified the `RefreshDearCacheBySubtasks` view to use a new boolean parameter `rebuild_in_days_to_refresh` for the Celery task.
- This replaces the previous method of passing the string `rebuild_period` directly, simplifying the logic for how the task determines whether to rebuild based on `days_to_refresh`.

The condition for sending the customer missing diagnosis email in `task_diagnose_missing_customers` was modified to always be true (from `total_missing_count > 0` to `total_missing_count > 0 or True`), likely for testing purposes.

- Updated log message for missing customers found to use plural "Customers".
- Added the total count of missing customers to the subject of the diagnostic email.

This commit introduces two key changes:

1.  **Refined Overlay Discovery (`amazon_eks/helpers/tui_deployer.py`):** The `find_overlays` function has been updated to explicitly restrict its search path to `amazon_eks/overlays`. This prevents the TUI deployer from processing retired or template overlay configurations, ensuring only active and intended configurations are considered during deployment.
2.  **Retired Overlay Version Update (`amazon_eks/retired_overlays/ortc/kustomization.yaml`):** The `django_api_sync` image version and its corresponding literal in the `version-configmap` have been updated for the `ortc` retired overlay. This is a maintenance update for a retired configuration.

Adds a graphical countdown timer and progress bar to the TUI deployment status bar. This replaces the static "Sleeping for 30s..." message with a dynamic visual countdown after each successful deployment step.

Refactored the `amazon_eks/helpers/deploy_all.sh` script to delegate its core logic to a new Python-based Text User Interface (TUI) script, `amazon_eks/helpers/tui_deployer.py`.

The changes include:
- **`amazon_eks/helpers/deploy_all.sh`**: Transformed into a simple launcher, now solely responsible for invoking the new `tui_deployer.py` script. All previous shell-based logic for finding overlays, iterating, and managing deployment steps has been removed.
- **`amazon_eks/helpers/tui_deployer.py` (new file)**: Implements the main deployment orchestration. This Python script provides:
    - A curses-based TUI for interactive deployment monitoring.
    - Automatic discovery and sorting of EKS overlay directories.
    - Sequential execution of `deploy_new_version.sh` for each overlay.
    - Real-time display of deployment logs within the TUI.
    - Progress indication, status messages, and colored feedback (success/failure).
    - Support for `start-from` functionality, mirroring the previous shell script.
    - Logging to `/tmp/tui_deployer.log` for debugging.
    - Error handling, pausing the deployment on failure.

No Django app names were identified from the paths of the modified files.

This commit refactors the `cached_dear` app by renaming the `make_all_dirty` parameter to `rebuild_in_days_to_refresh` across various functions, methods, and their calls. This change clarifies the parameter's intent, which controls whether a cache refresh performs a full rebuild for the specified `days_to_refresh` period (reloading every document) or uses a faster scan method.

The parameter renaming has been propagated to:
- `cached_dear/connector/dear_cached_api.py`: Updated method signatures, docstrings, internal calls, and variable names.
- `cached_dear/pricing_tasks.py`: Updated calls to `update_products`.
- `cached_dear/pricing_views.py`: Updated calls to `update_products`.
- `cached_dear/tasks.py`: Updated task signatures, periodic task configurations, and logic for cache refreshing. A new monthly cache refresh periodic task was also added.
- `project/common_code.py`: Updated an example in a comment.
- `shopify/tasks.py`: Updated calls to `update_products`.
- `zoho_crm/zoho_crm_business_logic/dear_to_zoho_quote_sync.py`: Updated calls to `refresh_dear_cache_by_subtasks`.

Additionally, `cached_dear/views_diagnostic.py` has been enhanced to include a direct link to the associated job view when displaying active lock information, improving traceability and debugging.

The `generate_changelog.sh` script now uses `%aI` instead of `%ar` in `git log` to display the author date in ISO 8601 format, providing absolute dates instead of relative ones.

**Summary of Changes:**
The `amazon_eks/helpers/generate_changelog.sh` script has been updated to include more commits in the generated deployment changelog. Specifically:
1.  The `git log` command now fetches the last 100 commits instead of 50.
2.  The corresponding message in the generated HTML changelog has been updated to reflect "Showing the last 100 commits included in this build."

This commit enhances the `amazon_eks/helpers/generate_changelog.sh` script by:

*   Increasing the number of displayed commits from 20 to 50.
*   Modifying the changelog table to include the full commit message (subject and body).
*   Adding CSS styling to display the commit body within a `<pre>` tag for better readability.
*   Improving the script's robustness by using null-byte delimiters (`-z`) with `git log` and a `while read -d ''` loop to safely parse multi-line commit messages and bodies, including basic HTML escaping.

**Django App(s) Impacted:** None (utility script in `amazon_eks/`)

Adds an automatic changelog generation process for deployments.
- A new script (`amazon_eks/helpers/generate_changelog.sh`) creates `core/static/core/changelog.html` based on the last 20 git commits, only regenerating if the current commit differs.
- This script is integrated into the `amazon_eks/helpers/deploy_new_version.sh` script.
- The `core/templates/admin/base_site.html` is updated to include a link to the generated changelog in the Django admin footer.
- The generated changelog file is added to `.gitignore`.

Removed unused `TypeAlias` import, added/removed blank lines for better readability, and corrected minor typos and formatting in comments across various Dear API TypedDicts. No functional changes.

Corrected the definition of `SaleCreditNoteModelEnhanced` to properly inherit from `SaleCreditNoteModel` instead of an incorrect string literal. This fixes a syntactic issue in the TypedDict definition.

This commit introduces several improvements and fixes across the `cached_dear` and `dear_zoho_analytics` Django apps.

**cached_dear:**
*   **Performance:** Implemented `.iterator()` for large queryset operations in `DearCache` and `ProductAvailability` models to reduce memory usage during full refreshes of stock adjustments, product availability, customer, order, sale credit, purchase, and supplier caches.
*   **Data Integrity:** Enhanced customer refresh logic to ensure all records are fetched from Dear API for `delete_not_seen` operations by using `MINIMUM_DATETIME` as the `modified_since_param`, ensuring proper deletion of stale records.
*   **Notifications:** Added email notification when stale customer records are deleted from the cache.
*   **Type Hinting:** Updated return type hint for `update_sale_credit_note_cache_with_sale` to use the new `SaleCreditNoteModelEnhanced` typed dict.
*   **Minor Fixes:** Corrected a log message typo from "orders" to "Purchases" for cache deletion. Updated `order_analysis_logic` type hint.

**cached_dear/dear_api_base:**
*   **API Model Updates:** Renamed `IsDeprecated` to `Deprecated` in the `Location` typed dict to match the Dear API field name.
*   **New Typed Dict:** Introduced `SaleCreditNoteModelEnhanced` to include an optional `LastModifiedOn` field, supporting cached functionality.

**dear_zoho_analytics:**
*   **Code Refactoring:** Moved `get_tax_discount` utility function out of `prepare_sales_fact_for_order` for better reusability and scope.
*   **Bug Fixes:** Corrected the retrieval of `document_exrate` for credit notes by fixing a typo (`CredtNoteConversionRate` to `CreditNoteConversionRate`) and ensuring it falls back to the general `CurrencyRate`.
*   **Abstraction:** Added abstract methods (`dear_date_string_backend_string_format`, `year_month_currency`, `get_max_modified_date`) to the `AnalyticsBackend` class, preparing for more flexible backend integrations.
```

The `cached_dear` app received updates to its Celery tasks:
- The `sync_pepperi_orders` task is now explicitly assigned to the `settings.CELERY_TASK_GEVENT_QUEUE` to ensure it runs on the appropriate Celery worker type.
- The `task_diagnose_missing_customers` task has been enhanced to:
    - Track which instances have missing customers.
    - Trigger an email notification to a predefined recipient (`tim@growthpath.com.au`) if any missing customers are found, including the names of the affected instances in the subject.

**cached_dear:**
*   **Enhancements:** Enhanced customer cache refresh (`update_customers`) to optionally delete stale records (not present in Dear API response) during full history synchronizations. Optimized `DearCache` updates by switching from `save()` to `update_or_create()`.
*   **Refactoring:** Removed `JobIdForm` and deprecated `cache_update_timestamp_sales_mini` view/URL.
*   **Migration:** Migrated `job_view` functionality (view, URL patterns, template `select_job.html`) to the `core` app, and updated relevant navigation links.
*   **Tasks:** Modified the `task_diagnose_missing_customers` schedule to run hourly and refined its log messages.

**core:**
*   **Job Viewing Centralization:** Implemented `job_view` functionality (previously in `cached_dear`), providing a centralized interface for viewing live job progress via websockets.
*   **Forms:** Introduced a more robust `JobIdForm` using `ModelChoiceField` to allow selection from recent jobs, and also supporting direct ID input.
*   **Admin Integration:** Added a "View Live Log" link in the Django Admin for `JobMaster` entries, linking directly to the live job progress page.
*   **Templates:** Added new templates (`generic_websocket_job.html`, `select_job.html`) for job viewing, and improved `websocket_include.html` with a shareable link for live progress.
*   **Models:** Enhanced the `JobMaster` string representation to include its status for better readability.
*   **UI:** Minor text refinement in the Django Admin footer.

Changes:
- Replaced hardcoded Celery task strings with `.name` attribute for improved robustness.
- Removed unused and commented-out logging configuration and task setup code.
- Updated the source for the promise dates health check URL from Django settings to environment variables.
- Added a second health check call after the promise date API task completes.