diff --git a/upgrade-plan/GEM_AUDIT.md b/upgrade-plan/GEM_AUDIT.md new file mode 100644 index 000000000..7c8d1755b --- /dev/null +++ b/upgrade-plan/GEM_AUDIT.md @@ -0,0 +1,79 @@ +# Gem Audit & Upgrade Risks + +This document summarizes the gems identified across the repository (`admin`, `web`, `ruby`, `websocket-gateway`, `monitor`, `pb`) that pose significant risks or require specific attention during the upgrade to Rails 8 and Ruby 3.x. + +**Status:** Draft / Initial Audit +**Date:** 2025-12-19 + +## 🚨 Critical / Blockers + +These gems are either incompatible with modern Rails/Ruby, have been removed, or require major architectural changes. + +| Gem | Current Version | Issue | Action Plan | +| :--- | :--- | :--- | :--- | +| **Ruby** | 2.3.1 | **Ancient**. EOL. Incompatible with Rails 8 (req 3.2+). | Upgrade to 3.2+ immediately. | +| **Rails** | 4.2.8 | **Ancient**. EOL. APIs removed/changed significantly. | Upgrade path: 4.2 -> 5.0 -> 5.1 -> 5.2 -> 6.0 -> 6.1 -> 7.0 -> 7.1 -> 8.0. | +| `protected_attributes` | * | Removed in Rails 5. | Replace with `StrongParameters` in all controllers/models. | +| `rails-observers` | * | Removed in Rails 4/5. | Replace with `ActiveSupport::Callbacks` or Service Objects. | +| `devise` | 3.3.0 | Incompatible with modern Rails. | Upgrade to latest (v4.x). meticulous migration of config/views required. | +| `activeadmin` | ~1.0.0.pre | Major breaking changes in v2/v3. | Upgrade to latest. Expect broken UI/configs. | +| `paperclip` / `carrierwave` | 0.x | `paperclip` is dead. `carrierwave` 0.x is very old. | Upgrade `carrierwave` to 3.x. Check config/storage adapters. | +| `aws-sdk` | ~> 1 | v1 is deprecated. v2/v3 have different namespaces. | Rewrite AWS interactions using `aws-sdk-s3`, etc. (v3). | +| `resque` ecosystem | Mixed | Old versions. `resque-scheduler`, `retry` issues. | Verify compatibility or migrate to **Solid Queue** (Rails 8 default) or Sidekiq. | +| `poltergeist` / `phantomjs` | * | **Dead project**. | Replace with `selenium-webdriver` (Headless Chrome) or `cuprite`. | +| `mimemagic` | * | Licensing issues/yanked versions. | Ensure using a safe version or replace with `marcel` (Rails default). | +| `json` | 1.8.6 | C-ext compilation fails on Ruby 3. | Remove pin. Use default or newer version. | + +## ⚠️ High Risk / Major Upgrades + +| Gem | Issue | Action | +| :--- | :--- | :--- | +| `omniauth` & strategies | API changes, security vulnerabilities. | Upgrade all `omniauth-*` gems. Verify provider APIs (FB, Google, Twitter, Stripe). | +| `paypal-sdk-*` / `recurly` / `stripe` | Payment processing. High risk of breaking revenue. | Audit APIs. Upgrade SDKs. **Test heavily**. | +| `elasticsearch` | Client and Server incompatibility. | Upgrade client gem. Ensure backend ES instance is compatible. | +| `react-rails` | v1.3.3 is very old. | Upgrade to latest. Verify React component integrations. | +| `sass-rails` / `sass` | Ruby Sass is EOL. | Migrate to `sassc-rails` or `dart-sass-rails` (preferred for Rails 8). | +| `jquery-rails` / `jquery-ui-rails` | Rails 8 drops jQuery dependency. | Manually include if needed, or refactor to Vanilla JS / Hotwire. | +| `coffee-rails` | Obsolete. | Convert `.coffee` to `.js` or modern ES6+. | +| `fog` / `fog-brightbox` | Monolithic gem split into parts. | Replace with specific `fog-*` gems or newer AWS SDKs. | +| `therubyracer` | V8 engine binding issues. | Remove. Use `mini_racer` or system Node.js via ExecJS. | + +## 🔍 Specific Ecosystems + +### Private / Local Gems +* `jam_ruby` (`ruby/`) +* `jampb` (`pb/`) +* `jam_websockets` (`websocket-gateway/`) +* **Action:** Ensure these local gems are compatible with Ruby 3 syntax (e.g., keyword arguments `**kwargs` vs `options` hash). + +### Database (PostgreSQL) +* Current: `pg` ~0.17 / 0.21. +* Target: `pg` ~1.5. +* **Action:** Remove OS-specific pins. Upgrade gem. Verify `postgres-copy`, `postgres_ext` compatibility. + +### Frontend / Assets +* **Current:** Sprockets 3, CoffeeScript, Sass, jQuery, Bootstrap 2(?). +* **Target:** Propshaft (Rails 8 default) or Sprockets 4. +* **Action:** Major frontend modernization likely needed. `jam-ui` seems newer/separate? + +### Testing +* `rspec-rails`: Upgrade to v6+. +* `factory_girl`: Renamed to `factory_bot`. Search/Replace globally. +* `database_cleaner`: Upgrade. +* `capybara-webkit`: Deprecated. Remove. + +## 📝 Other Notable Pins to Remove + +* `nokogiri` (Pinned 1.10.10) +* `sprockets` (Pinned 3.6.3) +* `faraday` (Pinned 0.9.2) +* `redis` (Pinned 3.3.3) +* `rubyzip` (Pinned 1.2.1) +* `icalendar` (Pinned 2.4.0) +* `email_validator` (Pinned 1.6.0) + +## Recommendation + +1. **De-risk Ruby:** Try to get the app booting on Ruby 2.7 first (bridge to 3.0), then 3.2. +2. **De-risk Rails:** Upgrade 4.2 -> 5.0 is the hardest hurdle (`protected_attributes`, `observers`). +3. **Component Isolation:** Upgrade `ruby` (shared lib) first, then `admin` (smaller footprint), then `web` (largest surface area). diff --git a/upgrade-plan/INIT.md b/upgrade-plan/INIT.md new file mode 100644 index 000000000..c70c74546 --- /dev/null +++ b/upgrade-plan/INIT.md @@ -0,0 +1,46 @@ +# We need to upgrade from Rails 4 to Rails 8.1.1 + +Upgrading from Rails 4 to 5 to 6 to 7 to 8 requires a careful plan. We need to 1st formulate the plan, and agree to it, before writing any code. + +**Latest Updates:** +* [Gem Audit & Risks](GEM_AUDIT.md) - A detailed breakdown of problematic gems and the upgrade strategy. + +Here's the current state of the project, however. + +Currently, we are using extremely old gems. We are depending on very old versions of Ruby. So old, that it is almost impossible to even build the a version of Ruby on a recent macOS machine. The code is deployed to an Ubuntu 12 machine, perhaps 10 years old by now. Finally, the postgresql instance is running 9.3.3. We don't really want to upgrade unless we have to. But if we do, we do. + +We *had* an extremely good set of both web tests, and typical model tests. The web tests generally do not work, because they relied on poltergeist and/or phantomjs, which stopped working a very long time ago. + +In a perfect world, all the tests passed, and we could upgrade from 4 to 5 to 7 to 8 Ruby on Rails, and the tests would work along the way. But this does not seem possible. So we need to, perhaps, just upgrade to 8, and then try to at least get all the tests to pass again. + +A few more details that are important that add some complexity. We actually have two active branches, both of which are deployed to servers. + +The `develop` branch is the older branch; it is deployed on 3 servers. It runs a working web interface that our older 'native client'. We have also been working on `promised_based_api_interation` branch, which has primarly only a new web interface supported, built in the `jam-ui` folder, only found on that branch. + +So, in general, if we add new rest API endpoint capabalities, we *should* add them to both `promised_based_api_interation` and `develop`, but we almost certainly have not consistently done that, and should check where there are differents and create a unified set of changes on the backend code. + +That said, let me describe the general structure of this code base: +* db - these used to be where database migrations existed, but this is now defunct. You should be able to ignore this folder for the most part. +* ruby - all models and 'library shared code' live here. This should be the same across both branches, but may not be. We should find the difference and try to reconcile before migrating the app. +* web - contains controllers (also, these should be the same across both branches, but may not be. We should find the difference and try to reconcile before migrating the app. ). For web UI though, the web assets/pipeline are active on the `develop` branch (still live and working for some of the user population, and unfortunately still important). The web/assets are not active for the `promised_based_api_interation` population, and instead the `jam-ui` web app is deployed as a cloudfront/static web site, and just uses the controllers. (rails as a backend, not a frontend; ) +* jam-ui - this is a static site; it is not a rails app, but uses the rails backend from web/ruby, but only from the `promised_based_api_interation` branch in production. + +So put it this way: +* a percentage of the user population uses the older web UI from the `develop` branch, and that frontend uses the same controllers and models also from that `develop` branch. These users are not at all aware of `jam-ui`. +* Another percentage of users do interface with `jam-ui`, and `jam-ui` uses controllers and models from `web` and `ruby` code bases, but only the `promised_based_api_interation`. +* websocket-gateway - luckily, we only use the version of the `develop` branch, for both the `develop` and `promised_based_api_interation` user base. +* luckily the database is also shared. +* admin - this is a backend, hidden admin portal used just by jamkazam employees. It needs to be upgradet too; luckily it's only active on rails 8. It is arguably easier to upgrade, however, it has only a few tests. We assume though we will have to upgrade both `admin` and `web` and `ruby` and `webscotke-gateway` all in lockstep, because all re-use code and library stemming from `ruby`; upgrade `ruby` , and upgrade them all. However, we could just upgrade everything, and start by trying to fix the tests first in `ruby` first, then `admin` and then in `websocket-gateway`, and finally `web`, since it has the most tests. + +Also, we use rabbitmq, but we use it in the most trivial way, and have not seemingly been impacted by upgrading it. + + +Our goal is to end up on one active branch `rails-8` where: + +* any small differences in `ruby` and `web` between both branches are reconciled (i.e., model differences and controller differences, primarily) +* The web UI is preserved and functioning 'as-is' in the develop branch (i.e., everything in web/app/views an web/assets/* still work) +* jam-ui works (it can still use the rest API successfully) +* By necessity, we are using a modern ruby. + +## Short-hand +In short-hand, we can call the `promised_based_api_interation` branch the `promise` branch. \ No newline at end of file diff --git a/upgrade-plan/MIGRATION.md b/upgrade-plan/MIGRATION.md new file mode 100644 index 000000000..1e2561bdf --- /dev/null +++ b/upgrade-plan/MIGRATION.md @@ -0,0 +1,38 @@ +# Migration Plan: Step 1 + +**Goal:** Establish a baseline and attempt the first major leap: Upgrading the shared `ruby` library to Ruby 3.2.2. + +## Phase 1: Verification (The "Before" State) + +Before changing anything, we must confirm the current environment works as expected on this machine. + +1. **Verify Web Boot:** + * Command: `cd web && bin/start_mx` + * Expectation: Rails server starts on port 3000. + * Action: Stop server (Ctrl+C) once "Listening on..." appears. + +2. **Verify Shared Lib Tests:** + * Command: `cd ruby && BUNDLE_GEMFILE=Gemfile.alt bundle _1.17.3_ exec rspec` + * *Note:* We use `Gemfile.alt` and Bundler 1.17.3 because that's what `web/bin/start_mx` suggests is the "working" configuration on this machine. + * Expectation: Tests run (pass or fail, but they *run*). + +## Phase 2: The Upgrade (The "After" State) + +We will target the `ruby` directory first. It is the foundation. If `ruby` doesn't work on Ruby 3.2, nothing else will. + +1. **Target:** Ruby 3.2.2 +2. **Files to Change:** `ruby/Gemfile` and `ruby/Gemfile.alt` +3. **Changes:** + * Update `ruby "..."` directive to `ruby "3.2.2"`. + * Remove hard pins on ancient gems (e.g., `nokogiri '1.10.10'`, `json '1.8.6'`). + * Update `pg` gem to remove OS-specific pins. +4. **Execution:** + * Command: `cd ruby && rbenv local 3.2.2 && bundle install` + * *Note:* We will use the modern Bundler provided with Ruby 3.2.2, not 1.17.3. +5. **Verification:** + * Command: `bundle exec rspec` + * Expectation: Dependencies resolve, and tests run. + +## Rollback Plan + +If we get stuck, we can simply `git checkout ruby/Gemfile*` to revert to the working state. diff --git a/upgrade-plan/PLAN.md b/upgrade-plan/PLAN.md new file mode 100644 index 000000000..53450eec3 --- /dev/null +++ b/upgrade-plan/PLAN.md @@ -0,0 +1,65 @@ +# Rails 8 Upgrade Plan + +## Objective +Upgrade the Jam Cloud codebase from Rails 4 / Ruby 2.3 to Rails 8.1 / Ruby 3.4. This process includes reconciling the `develop` branch (serving legacy Web UI and native clients) with the `promised_based_api_interation` branch (serving the new `jam-ui` React frontend). + +## Current State +- **Branches**: `develop` (Main), `promised_based_api_interation` (Promise). +- **Ruby**: 2.3.1 +- **Rails**: 4.2.8 +- **Components**: + - `ruby`: Shared models/libs. + - `web`: Controllers/Views (Legacy UI + API). + - `admin`: Admin portal. + - `jam-ui`: Static React site (depends on `web` API). + +## Strategy +We will perform a "Big Bang" upgrade on a new branch `rails-8`, starting with reconciliation, then a major jump in Ruby/Rails versions, followed by iterative fixing. + +## Phase 1: Preparation & Reconciliation +1. **Create Branch**: Create `rails-8` branch off `develop`. +2. **Merge Promise**: Merge `promised_based_api_interation` into `rails-8`. + - *Goal*: Combine new API capabilities from Promise with legacy UI from Develop. + - *Action*: Resolve merge conflicts in `ruby` (Models) and `web` (Controllers). + - *Reference*: See `RECONCILIATION_NOTES.md` for identified conflict areas. +3. **Sanity Check**: Ensure the merged codebase structurally looks correct (files from both branches are present). + +## Phase 2: Environment Upgrade +1. **Ruby Upgrade**: Update `.ruby-version` files to `3.3.0` (or latest stable). +2. **Gem Cleanout**: + - Remove version pins in `Gemfile`s that enforce old Ruby/Rails. + - Update `rails` gem to `~> 8.1.1`. + - Replace/Remove obsolete gems (`rails-observers` -> `rails-observers` gem compatible with 5+ or concern, `therubyracer` -> `mini_racer`, etc.). +3. **Bundle Update**: Attempt `bundle update` and resolve dependency graph conflicts. + +## Phase 3: Codebase Modernization (Fixing Boot) +1. **Configuration**: Update `config/application.rb` and initializers. Rails 8 defaults differ significantly. +2. **Boot Fixes**: Try to boot the app (`bin/rails c`). Fix immediate errors: + - Syntax changes (Ruby 2.3 -> 3.x keyword arguments). + - Removed Rails classes/methods (e.g., `ActiveRecord::Base.raise_in_transactional_callbacks`). + - Gem incompatibilities. + +## Phase 4: Application Logic Upgrade +1. **Models (`ruby`)**: + - Update ActiveRecord syntax (e.g., `scope` lambda requirement). + - Fix callbacks (if using Observers, migrate to Callbacks or separate gem). +2. **Controllers (`web` & `admin`)**: + - Update `ActionController` parameters (Strong Parameters should already be in Rails 4, but check strictness). + - Fix `render` calls if syntax changed. +3. **Assets**: + - Ensure `sprockets` or new asset pipeline works for Legacy UI. + - Verify `jam-ui` API endpoints are accessible. + +## Phase 5: Testing & Verification +1. **Fix Tests**: + - Start with `ruby` model tests. + - Move to `web` controller tests. + - (Note: Many web tests are broken (PhantomJS). Focus on Unit/Model tests and Request specs first). +2. **Manual Verification**: + - Spin up server. + - Check Legacy Web UI (login, dashboard). + - Check `jam-ui` (connect to local API). + +## Phase 6: Deployment Prep +1. **Docker/CI**: Update build scripts/Dockerfiles for Ruby 3.3. +2. **Migrations**: Ensure any pending migrations from both branches are consolidated. diff --git a/upgrade-plan/RECONCILIATION_NOTES.md b/upgrade-plan/RECONCILIATION_NOTES.md new file mode 100644 index 000000000..b41161e0f --- /dev/null +++ b/upgrade-plan/RECONCILIATION_NOTES.md @@ -0,0 +1,42 @@ +# Reconciliation Notes + +## Overview +We are merging `promised_based_api_interation` into `develop` (via new branch `rails-8`). + +## Common Ancestor +Commit: `245b39f67d05af5811ff7b84aa2b0c84d9cea02e` + +## Key Conflict Areas +The following files have been modified in both branches and will likely raise merge conflicts. + +### Shared Models (`ruby/lib/jam_ruby/models/`) +* `user.rb`: + * `develop`: Added `RECORDING_PREF_*` constants. + * `promise`: Added `JAM_REASON_PRESENT` to validation inclusion. + * *Resolution*: Keep both sets of additions. +* `active_music_session.rb` +* `feed.rb` +* `artifact_update.rb` +* `recording.rb` + +### Web Controllers (`web/app/controllers/`) +* `api_jam_tracks_controller.rb` +* `api_search_controller.rb` +* `artifacts_controller.rb` +* `landings_controller.rb` + +### Configuration (`web/config/`) +* `routes.rb`: Crucial. Likely new routes added in `promise` for API, and new routes in `develop` for features. + * *Resolution*: Merge carefully to preserve both sets of routes. +* `application.rb` + +### Assets +* `web/app/assets/javascripts/`: Significant divergence. `promise` has many new React components and stores. `develop` has maintenance updates. + +### Admin (`admin/`) +* `config/application.rb`: Both branches updated `musician_count`. No conflict expected. + +## Strategy +1. **Prioritize `develop` for Legacy UI**: Ensure changes to existing views/assets in `develop` take precedence if `promise` modified them (unlikely for views, likely for shared JS). +2. **Prioritize `promise` for New API**: Ensure new Controllers and Models methods required for `jam-ui` are preserved. +3. **Union of Features**: The `User` model example suggests features were added independently. We want the union of these features.