Support mapping of entry data to database columns (#273)

* Add upgrade note to readme

* Missing update script

* Correct file path to migration

* Lets not exclude parent

* Split forms and form submissions, provide Eloquent Submission Repository and Query Builder (#177)

* Add upgrade note to readme

* Begin split configs

* Use form handle to relate, rather then a database id

* Defer to core methods for retrieving submissions

* Update fresh migrations and add migration for existing installs

* Eloquent FormRepository

* Fix migration

* Dont force a model type or it can't be overwritten

* Missing update script

* Correct file path to migration

* Lets not exclude parent

* StyleCI

* Actually run update script

* Prep for Statamic 5

* Update dev dependencies

* Try again

* Meh

* Remove deprecated test functions

* static

* Migrate PHP unit config

* Fix migration

* Fix tests to use handle instead of id

* Bug fixes

* Fix form listing bugs

* Split migration publish tags (#266)

* Split migration tags

* fix everything

* Merge branch '5.x-support' into split-migration-tags

* Ignore `.phpunit.cache`

---------

Co-authored-by: Ryan Mitchell <[email protected]>

* Statamic 5: Simplify `TestCase` (#267)

* Simplify `TestCase`

* Remove `partialMock` method from `TestCase`

I'm not sure which "earlier versions of Laravel" this is referring to but the test suite seems to pass without it so I presume it's no longer needed.

* doesn't look like we're using this method either

* The `ConsoleKernel` isn't needed

* Make drivers opt-in, rather than opt-out (#268)

* Eloquent driver should be opt-in, not opt-out

* Swap all drivers to `eloquent` in our `TestCase`

* Drop status on entries (#228)

* Fix asset import bug

* Drop status on entries

* Revert "Fix asset import bug"

This reverts commit 35ebe65.

* Fix test

* StyleCI

* Revert

* Proper update script

* Make status nullable in down migration

* Statamic 5: Docs Refresh (#254)

* Initial docs refresh

* Fix tpyo

* Missing space

---------

Co-authored-by: Ryan Mitchell <[email protected]>

* Support mapped data columns

* Add tests and fix test errors by freezing time

* Update `AddonTestCase` import

* Fix failing tests in `EntryQueryBuilderTest` (#275)

* Fix failing tests in `EntryQueryBuilderTest`

* dont need to freeze time for this test

* 🍺

* 🍺

* Remove accidental testcase addition

* Avoid make()-ing

* Refactor out hooks

* Add test coverage after bringin data into COLUMNS

* Not array_keys

* Add assertion that it doesnt save the field to the data column

* 🍺

* Remove `@test` in favour of #[Test]

* Remove ::hook calls

* 🍺

---------

Co-authored-by: Duncan McClean <[email protected]>

Author: ryanmitchell

README.md

@@ -52,6 +52,10 @@ The configuration file, found in `config/statamic/eloquent-driver.php` is automa
 
 For each of the repositories, it allows you to determine if they should be driven by flat-files (`file`) or Eloquent (`eloquent`). Some repositories also have additional options, like the ability to override the model used.
 
+### Mapping Entry data
+
+If you want to map fields from your blueprints to columns with the same handle in your blueprint, set `entries.map_data_to_columns` to true. When adding new columns in a migration we recommend resaving all Entries so that column data is filled: `Entry::all()->each->save()`.
+
 ## Upgrading
 
 After updating to a new version of the Eloquent Driver, please ensure you run `php artisan migrate` to update your database to the latest schema.

config/eloquent-driver.php

@@ -39,6 +39,7 @@
         'driver' => 'file',
         'model' => \Statamic\Eloquent\Entries\EntryModel::class,
         'entry' => \Statamic\Eloquent\Entries\Entry::class,
+        'map_data_to_columns' => false,
     ],
 
     'fieldsets' => [

src/Entries/Entry.php

@@ -6,6 +6,7 @@
 use Illuminate\Support\Carbon;
 use Statamic\Contracts\Entries\Entry as EntryContract;
 use Statamic\Entries\Entry as FileEntry;
+use Statamic\Facades\Blink;
 use Statamic\Facades\Entry as EntryFacade;
 
 class Entry extends FileEntry
@@ -16,6 +17,10 @@ public static function fromModel(Model $model)
     {
         $data = isset($model->data['__localized_fields']) ? collect($model->data)->only($model->data['__localized_fields']) : $model->data;
 
+        foreach ((new self)->getDataColumnMappings($model) as $key) {
+            $data[$key] = $model->$key;
+        }
+
         $entry = (new static())
             ->origin($model->origin_id)
             ->locale($model->site)
@@ -93,6 +98,8 @@ public static function makeModelFromContract(EntryContract $source)
             $data->put('parent', (string) $parent->id);
         }
 
+        $dataMappings = (new self)->getDataColumnMappings(new $class);
+
         $attributes = [
             'origin_id' => $origin?->id(),
             'site' => $source->locale(),
@@ -101,12 +108,16 @@ public static function makeModelFromContract(EntryContract $source)
             'date' => $date,
             'collection' => $source->collectionHandle(),
             'blueprint' => $source->blueprint ?? $source->blueprint()->handle(),
-            'data' => $data->except(EntryQueryBuilder::COLUMNS),
+            'data' => $data->except(array_merge(EntryQueryBuilder::COLUMNS, $dataMappings)),
             'published' => $source->published(),
             'updated_at' => $source->lastModified(),
             'order' => $source->order(),
         ];
 
+        foreach ($dataMappings as $key) {
+            $attributes[$key] = $data->get($key);
+        }
+
         if ($id = $source->id()) {
             $attributes['id'] = $id;
         }
@@ -173,4 +184,17 @@ public function makeLocalization($site)
         return parent::makeLocalization($site)
             ->data($this->data());
     }
+
+    public function getDataColumnMappings(Model $model)
+    {
+        if (! config('statamic.eloquent-driver.entries.map_data_to_columns', false)) {
+            return [];
+        }
+
+        return Blink::once("eloquent-schema-{$model->getTable()}", function () use ($model) {
+            $schema = $model->getConnection()->getSchemaBuilder()->getColumnListing($model->getTable());
+
+            return collect($schema)->reject(fn ($value) => in_array($value, EntryQueryBuilder::COLUMNS))->all();
+        });
+    }
 }

src/Entries/EntryQueryBuilder.php

@@ -4,7 +4,9 @@
 
 use Illuminate\Support\Str;
 use Statamic\Contracts\Entries\QueryBuilder;
+use Statamic\Eloquent\Entries\Entry as EloquentEntry;
 use Statamic\Entries\EntryCollection;
+use Statamic\Facades\Blink;
 use Statamic\Facades\Collection;
 use Statamic\Facades\Entry;
 use Statamic\Query\EloquentQueryBuilder;
@@ -21,7 +23,7 @@ class EntryQueryBuilder extends EloquentQueryBuilder implements QueryBuilder
     private const STATUSES = ['published', 'draft', 'scheduled', 'expired'];
 
     const COLUMNS = [
-        'id', 'site', 'origin_id', 'published', 'slug', 'uri',
+        'id', 'site', 'origin_id', 'published', 'slug', 'uri', 'data',
         'date', 'collection', 'created_at', 'updated_at', 'order', 'blueprint',
     ];
 
@@ -117,7 +119,9 @@ protected function column($column)
             $column = 'origin_id';
         }
 
-        if (! in_array($column, self::COLUMNS)) {
+        $columns = Blink::once('eloquent-entry-data-column-mappings', fn () => array_merge(self::COLUMNS, (new EloquentEntry)->getDataColumnMappings($this->builder->getModel())));
+
+        if (! in_array($column, $columns)) {
             if (! Str::startsWith($column, 'data->')) {
                 $column = 'data->'.$column;
             }

tests/Assets/AssetTest.php

@@ -4,6 +4,7 @@
 
 use Illuminate\Foundation\Testing\RefreshDatabase;
 use Illuminate\Support\Facades\Storage;
+use PHPUnit\Framework\Attributes\Test;
 use Statamic\Facades;
 use Tests\TestCase;
 
@@ -40,7 +41,7 @@ public function setUp(): void
         Facades\Asset::make()->container('test')->path('f.jpg')->save();
     }
 
-    /** @test */
+    #[Test]
     public function saving_an_asset_clears_the_eloquent_blink_cache()
     {
         $asset = Facades\Asset::find('test::f.jpg');

tests/Data/Entries/EntryQueryBuilderTest.php

@@ -190,6 +190,7 @@ private function createWhereDateTestEntries()
         EntryFactory::id('5')->slug('post-5')->collection('posts')->data(['title' => 'Post 5', 'test_date' => null])->create();
     }
 
+    #[Test]
     public function entries_are_found_using_where_null()
     {
         EntryFactory::id('1')->slug('post-1')->collection('posts')->data(['title' => 'Post 1', 'text' => 'Text 1'])->create();
@@ -796,6 +797,46 @@ public function entries_can_be_ordered_by_an_date_json_field()
         $this->assertEquals(['Post 2', 'Post 1', 'Post 3'], $entries->map->title->all());
     }
 
+    #[Test]
+    public function entries_can_be_ordered_by_a_mapped_data_column()
+    {
+        config()->set('statamic.eloquent-driver.entries.map_data_to_columns', true);
+
+        \Illuminate\Support\Facades\Schema::table('entries', function ($table) {
+            $table->string('foo', 30);
+        });
+
+        Collection::make('posts')->save();
+        EntryFactory::id('1')->slug('post-1')->collection('posts')->data(['title' => 'Post 1', 'foo' => 2])->create();
+        EntryFactory::id('2')->slug('post-2')->collection('posts')->data(['title' => 'Post 2', 'foo' => 3])->create();
+        EntryFactory::id('3')->slug('post-3')->collection('posts')->data(['title' => 'Post 3', 'foo' => 1])->create();
+
+        $entries = Entry::query()->where('collection', 'posts')->orderBy('foo', 'desc')->get();
+
+        $this->assertCount(3, $entries);
+        $this->assertEquals(['Post 2', 'Post 1', 'Post 3'], $entries->map->title->all());
+    }
+
+    #[Test]
+    public function entries_can_be_queried_by_a_mapped_data_column()
+    {
+        config()->set('statamic.eloquent-driver.entries.map_data_to_columns', true);
+
+        \Illuminate\Support\Facades\Schema::table('entries', function ($table) {
+            $table->string('foo', 30);
+        });
+
+        Collection::make('posts')->save();
+        EntryFactory::id('1')->slug('post-1')->collection('posts')->data(['title' => 'Post 1', 'foo' => 2])->create();
+        EntryFactory::id('2')->slug('post-2')->collection('posts')->data(['title' => 'Post 2', 'foo' => 3])->create();
+        EntryFactory::id('3')->slug('post-3')->collection('posts')->data(['title' => 'Post 3', 'foo' => 1])->create();
+
+        $entries = Entry::query()->where('collection', 'posts')->where('foo', 3)->get();
+
+        $this->assertCount(1, $entries);
+        $this->assertEquals(['Post 2'], $entries->map->title->all());
+    }
+
     #[Test]
     public function filtering_using_where_status_column_writes_deprecation_log()
     {
@@ -884,4 +925,15 @@ public static function filterByStatusProvider()
             ]],
         ];
     }
+
+    #[Test]
+    public function entries_are_found_using_where_data()
+    {
+        $this->createDummyCollectionAndEntries();
+
+        $entries = Entry::query()->where('data->title', 'Post 1')->orWhere('data->title', 'Post 3')->get();
+
+        $this->assertCount(2, $entries);
+        $this->assertEquals(['Post 1', 'Post 3'], $entries->map->title->all());
+    }
 }

tests/Entries/EntryTest.php

@@ -241,4 +241,64 @@ public function it_propagates_origin_date_to_descendent_models()
 
         $this->assertEquals($entry->descendants()->get('fr')->model()->date, '2024-01-01 00:00:00');
     }
+
+    #[Test]
+    public function it_stores_and_retrieves_mapped_data_values()
+    {
+        config()->set('statamic.eloquent-driver.entries.map_data_to_columns', true);
+
+        $collection = Collection::make('blog')->title('blog')->routes([
+            'en' => '/blog/{slug}',
+        ])->save();
+
+        \Illuminate\Support\Facades\Schema::table('entries', function ($table) {
+            $table->string('foo', 30);
+        });
+
+        $entry = (new Entry())
+            ->collection('blog')
+            ->slug('the-slug')
+            ->data([
+                'foo' => 'bar',
+            ]);
+
+        $entry->save();
+
+        $this->assertEquals('bar', $entry->model()->toArray()['foo']);
+        $this->assertArrayNotHasKey('foo', $entry->model()->data);
+
+        $fresh = Entry::fromModel($entry->model()->fresh());
+
+        $this->assertSame($entry->foo, $fresh->foo);
+    }
+
+    #[Test]
+    public function it_doesnt_store_mapped_data_when_config_is_disabled()
+    {
+        config()->set('statamic.eloquent-driver.entries.map_data_to_columns', false);
+
+        $collection = Collection::make('blog')->title('blog')->routes([
+            'en' => '/blog/{slug}',
+        ])->save();
+
+        \Illuminate\Support\Facades\Schema::table('entries', function ($table) {
+            $table->string('foo', 30)->nullable();
+        });
+
+        $entry = (new Entry())
+            ->collection('blog')
+            ->slug('the-slug')
+            ->data([
+                'foo' => 'bar',
+            ]);
+
+        $entry->save();
+
+        $this->assertNull($entry->model()->toArray()['foo']);
+        $this->assertArrayHasKey('foo', $entry->model()->data);
+
+        $fresh = Entry::fromModel($entry->model()->fresh());
+
+        $this->assertSame($entry->foo, $fresh->foo);
+    }
 }