``accessors``: convenience methods for accessing database entries

``populate``: convenience methods for populating database tables

The ``accessors`` and ``populate`` submodules are each split into ``schema`` and ``fts`` method
groups. The former concerns methods relating to the actual database schema, the latter to
their SQLite FTS counterparts.

 method

Subpackages are broken up by inheritance. Within a given submodule, you have a
``_base.py`` file defining the base class associated with that submodule's title, along
with concrete subclasses of that base in their own files. Deeper inheritance would
recursively extend this structure. The ``__init__.py`` for a given submodule then
exposes the concrete instances, leaving the base hidden. For example,

.. code-block::

    accessors/
        _base.py
        core.py
        fts.py

``core`` and ``fts`` house the ``CoreAccessor`` and ``FTSAccessor`` classes, respectively,
and are the direct subclasses of the ``Accessor`` parent found in the ``_base``. This base
class *could* be placed outside of the submodule in the parent directory (imported
with something like ``from db import accessor`` instead of ``from db.accessor import
_base``). This is entirely valid, but I tend to prefer when the base class is among its
direct children, as

- In this case at least, the base doesn't need to be exposed
- The base class is being stowed away under an appropriately named submodule; having a
  separate ``accessor.py`` and ``accessors/`` file/directory can feel a little cluttered.
- It makes imports across the accessors feel standardized:

  .. code-block:: python

      from localsys.db.accessors._base import Accessor

      from localsys.db.accessors.core import CoreAccessor

  Both have the same level of nesting to reach the class.

Frankly, both means of organization are perfectly fine, and as far as I can tell,
semantically sound in their own right. This particular scheme is just a preference in
the moment, and so long as I keep things consistent, choosing one over the other
shouldn't matter.

Additionally, note how ``__init__.py``s are typically set up when providing wider access
to internal modules. The ``init`` typically pulls out classes from sibling modules
(i.e., files), but will import subpackages are the topmost level. For example, for the
structure

    db/
        __init__.py
        accessors/
            __init__.py
            _base.py
            core.py
            fts.py

we have

   :name: db/__init__.py

    from localsys.db import accessors

which just imports the subpackage ``accessors``. However, within subpackage:

   :name: db/accessors/__init__.py

    from localsys.db.accessors.core import CoreAccessor

we don't just import the submodule ``core``; we did into the file to grab the relevant
class and pull it into the outer namespace. Overarching point: ``__init__.py`` files
typically reach into the sibling files (submodules) and pull out classes. Given that
this behavior is recursive, ``__init__.py`` then respect subpackages (nested
directories), importing them at the top-level and expecting an internal ``__init__.py``
will have managed access appropriately.

 file defining the base class associated with that submodule’s title, along
recursively extend this structure. The 

 for a given submodule then

 be placed outside of the submodule in the parent directory (imported
with something like 

). This is entirely valid, but I tend to prefer when the base class is among its

The base class is being stowed away under an appropriately named submodule; having a
separate ``accessor.py`` and ``accessors/`` file/directory can feel a little cluttered.

separate 

It makes imports across the accessors feel standardized:

.. code-block:: python

    from localsys.db.accessors._base import Accessor

Both have the same level of nesting to reach the class.

from localsys.db.accessors._base import Accessor

from localsys.db.accessors.core import CoreAccessor

shouldn’t matter.

structure

``__init__.py``s are typically set up when providing wider access
to internal modules. The ``init``

__init__.py``s are typically set up when providing wider access
to internal modules. The ``init

 typically pulls out classes from sibling modules
structure

db/
    __init__.py
        _base.py
        core.py
        fts.py

; we did into the file to grab the relevant
class and pull it into the outer namespace. Overarching point: 

 files
this behavior is recursive, 

 then respect subpackages (nested
directories), importing them at the top-level and expecting an internal 

At a glance, the organization of subpackages here feels like it clashes with those
seen in ``localsys.primitives``. ``note_components``, for instance, houses the components
for the outer ``note`` module. Contrast this with how the ``core`` submodule looks: it's
composing ``*/core.py`` files across subpackages ``accessors`` and ``managers``, rather than
a single subpackage like ``note``. This seems inconsistent, but the subpackages here are
actually still organized in the same way: by inheritance. It just happens that the
all of the note components inherit from the same base class, and are thus confined to
a single subpackage. This aside, the subpackages themselves are still created around
inheritance, wrapping up a base and direct subclasses.

seen in 

. This seems inconsistent, but the subpackages here are

Provides access to an underlying schema through a supported set of operations. Class
methods could be general, high-level SQL wrappers, or convenience functions for common
schema-specific queries.

Access wrapper class for complex queries and easy integration with Composer tables.
Implements high-level access to things like common constrained SELECT queries.

engine -- SQLAlchemy engine to use for queries. Engine is initialized dynamically as
a property (based on the config) if not provided

SQLAlchemy engine to use for queries. Engine is initialized dynamically as

Note that subclasses in this subpackage are split differently to other subpackages in the
DB. Instead of being split by table group, corresponding to a Composer (which defines that
table group), Accessors are split by a separate dimension: table "type". This is why we
have a "TableAccessor" and an "FTSAccessor": the former exposes access operations
available to generic tables, the latter to FTS tables (instead of being designed
specifically around "core" and "fts" groups, for instance).

table group), Accessors are split by a separate dimension: table “type”. This is why we
have a “TableAccessor” and an “FTSAccessor”: the former exposes access operations
specifically around “core” and “fts” groups, for instance).

Seeing as FTS tables are "generic" tables, it seems inconsistent not to have FTSAccessor
inherit from TableAccessor. While this would work fine, the model we're working with
doesn't really need it; you can instead think of the FTSAccessor as defining _only_
FTS-specific operations. Given that you have a Composer for your desired table group, you
can then wrap it with your desired set of "access actions," available in separate Accessor
subclasses.

Seeing as FTS tables are “generic” tables, it seems inconsistent not to have FTSAccessor
inherit from TableAccessor. While this would work fine, the model we’re working with
doesn’t really need it; you can instead think of the FTSAccessor as defining _only_
can then wrap it with your desired set of “access actions,” available in separate Accessor

For instance, you could wrap an FTSComposer in either a TableAccessor or FTSAccessor. The
former will treat the tables in the composer like regular tables, exposing methods like
``.select`` and ``.select_one``, whereas the latter defines FTS-specific actions like
``.search``.

FTSAccessor.search(table_name, select_cols='*', search_cols=None, q=None, colq=None, snip_col=0, hl_col=0, limit=100, snip=64, tokenizer='unicode61', group_by=None, agg_cols=None, wherein_dict=None, unique_on=None)

table_name, select_cols='*', search_cols=None, q=None, colq=None, snip_col=0, hl_col=0, limit=100, snip=64, tokenizer='unicode61', group_by=None, agg_cols=None, wherein_dict=None, unique_on=None

Execute a search query against an indexed FTS table for specific primitives. This
method is mostly a generic FTS handler, capable of handling queries to any available
FTS table with a matching naming scheme (``fts_<type>_<tokenizer>``). The current
intention is support all tokenizers, for file, note, block, and link primitives.

FTS table with a matching naming scheme (

). The current

Search results include all FTS table columns, as well as SQLite-supported ``snippet``s
and ``highlight``s for matches. Matches are filtered and ordered by SQLite's
``MATCH``-based score for the text & column queries. Results are (a list of) fully
expanded dictionaries housing column-value pairs.

Search results include all FTS table columns, as well as SQLite-supported 

``snippet``s
``MATCH``

snippet``s
``MATCH

-based score for the text & column queries. Results are (a list of) fully

GROUP BY cannot be paired with SQLITE FTS extensions; thus, we perform manual
group checks on the result set in Python before response

Analysis:
The returned JSON structure has been (loosely) optimized for speed on the client
side. Fully forming individual dictionary based responses saves time in
Javascript, as the JSON parser is expected to be able to create the objects
faster than post-hoc construction in JS. This return structure was compared
against returning an array of arrays (all ordered in the same fashion), along with
a column list to be associated with each of the result values. While this saves
some size on the payload (the same column names don't have to be transmitted for
each result), the size of the returned content massively outweighs the
predominantly short column names. The only way this structure would be viable is
if a significant amount was saved on transfer compared to the slow down in JS
object construction; this is (almost) never the case.

some size on the payload (the same column names don’t have to be transmitted for

search_cols (str | None) -- space separated string of columns to use for primary queries

colq (str | None) -- column constraint string; must conform to SQLite standards (e.g.,
<col>:<text>

column constraint string; must conform to SQLite standards (e.g.,

snip_col (int | None) -- table column to use for snippets (default: 1; source content column)

table column to use for snippets (default: 1; source content column)

hl_col (int | None) -- table column to use for highlights (default: 2; format column, applied
to HTML targets)

table column to use for highlights (default: 2; format column, applied

limit (int | None) -- maximum number of results to return in the SQL query

tokenizer (str | None) -- tokenizer to use (assumes relevant FTS table has been built)

wherein_dict (dict | None) -- (col-name, value-list) pairs to match result set against, via
WHERE ... IN clauses

(col-name, value-list) pairs to match result set against, via
WHERE … IN clauses

Dictionary with search results (list of column indexed dictionaries) and relevant
metadata.

One particular feature not supported by the current type hierarchy is the possible use of
different backends to implement a general interface like SQLAccessor. One could imagine,
for instance, using ``sqlalchemy`` or ``sqlite`` to define the same methods laid out in a
parent class blueprint. It's not too difficult to imagine cases where both of these may be
useful, but for now it is outside the development scope. Should it ever enter the scope,
however, we might consider a simple ``backend`` argument on instantiation, keeping just the
SQLAccessor exposed rather than a whole set of backend-specific types:

for instance, using 

 to define the same methods laid out in a
parent class blueprint. It’s not too difficult to imagine cases where both of these may be
however, we might consider a simple 

 argument on instantiation, keeping just the

class SQLAlchemyAccessor(RelationalAccessor): # may also inherit from a dedicated interface parent
    def select(...):
        ...

class SQLiteAccessor(RelationalAccessor):
        ...

class SQLAccessor(RelationalAccessor):
    backends = {
        'sqlalchemy': SQLAlchemyAccessor,
        'sqlite':     SQLteAccessor,
    }

    def __init__(self, backend: str):
        self.backend = self.backends.get(backend)

        return self.backend.select(...)

For now, we can look at SQLAccessor (and equivalents in other type hierarchies, like
SQLManagers) as being SQLAlchemyAccessors and not supporting any backend swapping. But in
theory, to make the above change, we'd just need to rename it and wrap it up.

theory, to make the above change, we’d just need to rename it and wrap it up.

Bases: :py:class:`~co3.accessor.Accessor`, :py:class:`~typing.Generic`

RelationalAccessor.select(connection, relation, attributes=None, where=None, distinct_on=None, order_by=None, limit=0, mappings=False, include_cols=False)

connection, relation, attributes=None, where=None, distinct_on=None, order_by=None, limit=0, mappings=False, include_cols=False

RelationalAccessor.select_one(connection, relation, attributes=None, where=None, mappings=False, include_cols=False)

connection, relation, attributes=None, where=None, mappings=False, include_cols=False

SQLAccessor.raw_select(connection, sql, bind_params=None, mappings=False, include_cols=False)

connection, sql, bind_params=None, mappings=False, include_cols=False

Parse SQLAlchemy results into Python dicts. Leverages mappings to associate full
column name context.

If ``query_cols`` is provided, their implicit names will be used for the keys of the
returned dictionaries. This information is not available under CursorResults and thus
must be provided separately. This will yield results like the following:

 is provided, their implicit names will be used for the keys of the

which can make accessing certain results a little more intuitive.

SQLAccessor.select(connection, component, columns=None, where=None, distinct_on=None, order_by=None, limit=0, mappings=False, include_cols=False)

connection, component, columns=None, where=None, distinct_on=None, order_by=None, limit=0, mappings=False, include_cols=False

Perform a SELECT query against the provided table-like object (see
``check_table()``).

Deprecated: String aliases
String aliases for tables are no longer supported. This method no longer checks
against any specific schema table-maps or Composers. Instead, this should be
done outside the Accessor.

group_by -- list of columns to group by; for now serves as a proxy for DISTINCT
(no aggregation methods accepted)

list of columns to group by; for now serves as a proxy for DISTINCT

order_by -- column to order results by (can use <col>.desc() to order
by descending)

column to order results by (can use <col>.desc() to order

Statement results, either as a list of 1) SQLAlchemy Mappings, or 2) converted
dictionaries

VSSAccessor.search(query, index_name, limit=10, score_threshold=0.5)

CO3 is an abstract base class for scaffolding object hierarchies and managing operations
with associated database schemas. It facilitates something like a "lightweight ORM" for
classes/tables/states with fixed transformations of interest. The canonical use case is
managing hierarchical document relations, format conversions, and syntactical components.

with associated database schemas. It facilitates something like a “lightweight ORM” for

Collate: organize and transform conversion outputs, possibly across class components

Collect: gather core attributes, conversion data, and subcomponents for DB insertion

Compose: construct object-associated DB table references through the class hierarchy

Group keys are simply named collections to make it easy for storage components to
be attached to action subsets. They do _not_ augment the action registration
namespace, meaning the action key should still be unique; the group key is purely
auxiliary.

Action methods can also be attached to several groups, in case there is
overlapping utility within or across schemas or storage media. In this case, it
becomes particularly critical to ensure registered ``collate`` methods really are
just "gathering results" from possibly heavy-duty operations, rather than
performing them when called, so as to reduce wasted computation.

becomes particularly critical to ensure registered 

 methods really are
just “gathering results” from possibly heavy-duty operations, rather than

Method to define how a subtype's inserts should be handled under ``collect`` for
canonical attributes, i.e., inserts to the type's table.

Method to define how a subtype’s inserts should be handled under 

Return "connective" collation component data, possibly dependent on
instance-specific attributes and the action arguments. This is typically the
auxiliary structure that may be needed to attach to responses from registered
``collate`` calls to complete inserts.

Return “connective” collation component data, possibly dependent on

Note: this method is primarily used by ``Mapper.collect()``, and is called just
prior to collector send-off for collation inserts and injected alongside collation
data. Common structure in collation components can make this function easy to
define, independent of action group for instance.

, and is called just

constituent components that need handling.

This module is the critical "middleware" connecting the primitive object definitions and
their representations in the database. It operates with full knowledge of how both are
defined, and abstracts away both the prep work for DB insertions as well as updates
trickling down the primitive hierarchy.

This module is the critical “middleware” connecting the primitive object definitions and

The ``src`` format target is re-used for both canonical tables/primitives, as well as
``<prim>_conversion_matter`` tables in tables/conversions under the ``src`` format. The
latter is meant to extend those attributes that are format-specific (i.e., would change
when, say, converting to ``html5``), and thus need to be broken across the format
dimension.

 format target is re-used for both canonical tables/primitives, as well as

 format. The
when, say, converting to 

Despite the structure of the database module, this class does not currently inherit
from a super class in localsys.db (like the accessors and managers, for instance).
This will likely ultimately be the model that's embraced, but until FTS (or other
groups) need a collector, this will be remain an independent class. It is, however,
named like a concrete subclass, taking on the "Core" prefix.

This will likely ultimately be the model that’s embraced, but until FTS (or other
named like a concrete subclass, taking on the “Core” prefix.

component (TypeVar(C, bound= Component)) -- Component from registered schema

insert_data (dict) -- dict with (possibly raw/incomplete) insert data

receipts (list | None) -- optional list to which generated receipt should be appended.
Accommodates the common receipt list aggregation pattern.

optional list to which generated receipt should be appended.

Collect insert-ready dictionaries for the core primitive schema. This method is
effectively a light wrapper around the File and Note-based collection logic
elsewhere in the class.

The overall collection scheme embraces a session-like sequential update model to
an internal insert tracker. The sequence of insert methods is ordered according to
the schema hierarchy, and higher level inserts dictate the scope for lower level
inserts (all methods check and populate the same ``inserts`` dictionary). Calling
this method flushes any existing inserts, ensuring a re-scan takes place across
calls (or "sessions").

inserts (all methods check and populate the same 

 dictionary). Calling
calls (or “sessions”).

skip_updated -- whether to ignore primitives with existing up-to-date
database entries

whether to ignore primitives with existing up-to-date

Table name-indexed dictionary of insert lists (of column name-indexed dicts)

General wrapper for storage components to be used in various database contexts. Relations
can be thought of generally as named data containers/entities serving as a fundamental
abstractions within particular storage protocols.

Dev note:
Any reason to have ComposeableComponents and Relations as separate types? The thought
is that there may be some possible Component types we want to be able to Compose that
wouldn't logically be Relations. But the gap here might be quite small

wouldn’t logically be Relations. But the gap here might be quite small

Bases: :py:class:`~co3.component.Component`, :py:class:`~typing.Generic`

Relation base for tabular components to be used in relation DB settings. Attempts to
adhere to the set-theoretic base outlined in the relational model [1]. Some
terminology:

Relation: table-like container
| -> Heading: set of attributes
|    | -> Attribute: column name
| -> Body: set of tuples with domain matching the heading
|    | -> Tuple: collection of values

Note: development tasks
As it stands, the Relation skeleton is incredibly lax compared to the properties and
operations that should be formally available, according its pure relational algebra
analog.

Relations are also generic up to a type T, which ultimately serves as the base object
for Relation instances. We aren't attempting to implement some generally useful
table-like class here; instead we're just exposing a lightweight interface that's
needed for a few CO3 contexts, and commonly off-loading most of the heavy-lifting to
true relation objects like SQLAlchemy tables.

analog.

for Relation instances. We aren’t attempting to implement some generally useful
table-like class here; instead we’re just exposing a lightweight interface that’s

Note that the sa.Table type is intentional here; not all matching types for
SQLTableLike have a defined ``name`` property

SQLTableLike have a defined 

include_all -- whether to include all columns, even those without explicit defaults

whether to include all columns, even those without explicit defaults

Central object for defining storage protocol-specific interfaces. The database wraps up
central items for interacting with database resources, namely the Accessor and Manager
objects.

The Database type hierarchy attempts to be exceedingly general; SQL-derivatives should
subclass from the RelationalDatabase subtype, for example, which itself becomes a new
generic via a type dependence on Relation.

While relying no many constituent pieces, Databases intend to provide all needed objects
under one roof. This includes the Engine (opens up connections to the database), Accessors
(running select-like queries on DB data), Managers (updating DB state with sync
insert-like actions), and Indexers (systematically caching Accessor queries). Generalized
behavior is supported by explicitly leveraging the individual components. For example,

with db.engine.connect() as connection:
    db.access.select(
        connection,
        <query>
    )
    db.manager.insert(
        component,
        data
    )

The Database also supports a few directly callable methods for simplified interaction.
These methods manage a connection context internally, passing them through the way they
might otherwise be handled explicitly, as seen above.

Older models supported Accessors/Managers that housed their own Engine instances, and
when performing actions like ``insert``, the Engine would be passed all the way through
until a Connection could be spawned, and in that context the single action would be
made. This model forfeits a lot of connection control, preventing multiple actions
under a single connection.

The newer model now avoids directly allowing Managers/Accessors access to their own
engines, and instead they expose methods that explicitly require Connection objects.
This means a user can invoke these methods in their own Connection contexts (seen
above) and group up operations as they please, reducing overhead. The Database then
wraps up a few single-operation contexts where outer connection control is not needed.

when performing actions like 

, the Engine would be passed all the way through

Generic to both a Component (C), and an Engine resource type (R). The Engine's
generic openness must be propagated here, as it's intended to be fully abstracted away
under the Database roof. Note that we cannot explicitly use an Engine type in its
place, as it obscures its internal resource type dependence when we need it for
hinting here in ``__init__``.

Generic to both a Component (C), and an Engine resource type (R). The Engine’s
generic openness must be propagated here, as it’s intended to be fully abstracted away
hinting here in 

Decide on official ruling for assigning Schema objects, and verifying any
attempted Component-based actions (e.g., inserts, selects) to belong to or be a
composition of Components within an attached Schema. Reasons for: helps complete
the sense of a "Database" here programmatically, incorporating a more structurally
accurate representation of allowed operations, and prevent possible attribute and
type collisions. Reasons against: generally not a huge concern to align Schemas as
transactions will rollback, broadly increases a bit of bulk, and users often
expected know which components belong to a particular DB. Leaning more to **for**,
and would only apply to the directly supported method passthroughs (and thus would
have no impact on independent methods like ``Accessor.raw_select``). Additionally,
even if component clashes don't pose serious risk, it can be helpful to
systematically address the cases where a misalignment is occurring (by having
helpful ``verify`` methods that can be ran before any actions).

the sense of a “Database” here programmatically, incorporating a more structurally
expected know which components belong to a particular DB. Leaning more to 

,
have no impact on independent methods like 

). Additionally,
even if component clashes don’t pose serious risk, it can be helpful to
helpful 

engine_args -- positional arguments to pass on to the Engine object during
instantiation

positional arguments to pass on to the Engine object during

engine_kwargs -- keyword arguments to pass on to the Engine object during

keyword arguments to pass on to the Engine object during

Variables:
_local_cache: a database-local property store for ad-hoc CacheBlock-esque
              methods, that are nevertheless _not_ query/group-by responses to
              pass on to the Indexer. Dependent properties should write to the
              this cache and check for existence of stored results; the cache
              state must be managed globally.

methods, that are nevertheless _not_ query/group-by responses to
pass on to the Indexer. Dependent properties should write to the
this cache and check for existence of stored results; the cache
state must be managed globally.

Accessing ``.manage`` queues a cache clear on the external index, as well wipes the
local index.

 queues a cache clear on the external index, as well wipes the

args and kwargs have to be general/unspecified here due to the possible
passthrough method adopting arbitrary parameters in subtypes. I could simply
overload this method in the relevant inheriting DBs (i.e., by matching the
expected Accessor's .select signature).

expected Accessor’s .select signature).

Bases: :py:class:`~co3.database.Database`, :py:class:`~typing.Generic`

accessor/manager assignments satisfy supertype's type settings;
``TabluarAccessor[Self, C]`` is of type ``type[RelationalAccessor[Self, C]]``
(and yes, ``type[]`` specifies that the variable is itself being set to a type or a
class, rather than a satisfying _instance_)

accessor/manager assignments satisfy supertype’s type settings;

 specifies that the variable is itself being set to a type or a

Engine base class. Used primarily as a Database connection manager, with general
methods that can be defined for several kinds of value stores.

Note that this is where the connection hierarchy is supposed to stop. While some
derivative Engines, like SQLEngine, mostly just wrap another engine-like object, this
is not the rule. That is, inheriting Engine subtypes shouldn't necessarily expect to
rely on another object per se, and if such an object is required, _this_ is the class
is meant to be skeleton to supports its creation (and not merely a wrapper for some
other type, although it may appear that way when such a type is in fact readily
available).

is not the rule. That is, inheriting Engine subtypes shouldn’t necessarily expect to

More specifically, why not just have all the functionality here packed into the
Database by default? The answer is that, realistically, it could be. The type
separation between the Engine and Database is perhaps the least substantiated in
CO3. That being said, it still serves a purpose: to make composition of subtypes
easier. The Engine is a very lightweight abstraction, but some Engine subtypes
(e.g., FileEngines) may be used across several sibling Database types. In this
case, we'd have to repeat the Engine-related functionality for such sibling types.
Depending instead on a singular, outside object keeps things DRY. If Databases and
Engines were uniquely attached type-wise 1-to-1 (i.e., unique Engine type per
unique Database type), a separate object here would indeed be a waste, as is the
case for any compositional typing scheme.

case, we’d have to repeat the Engine-related functionality for such sibling types.

This class is now non-generic. It was originally conceived as a generic, depending
on a "resource spec type" to be help define expected types on initialization.
This simply proved too messy, required generic type propagation to the Database
definition, and muddied the otherwise simple args and kwargs forwarding for
internal manager creation.

on a “resource spec type” to be help define expected types on initialization.

Open a connection to the database specified by the resource. Exactly what the
returned connection looks like remains relatively unconstrained given the wide
variety of possible database interactions. This function should be invoked in
with-statement contexts, constituting an "interaction session" with the database
(i.e., allowing several actions to be performed using the same connection).

with-statement contexts, constituting an “interaction session” with the database

Return Engine's singleton manager, initializing when the first call is made.

Return Engine’s singleton manager, initializing when the first call is made.

SQLEngine.exec_explicit(connection, statement, bind_params=None)

SQLEngine.execute(connection, statement, bind_params=None, include_cols=False)

Execute a general SQLAlchemy statement, optionally binding provided parameters and
returning associated column names.

CacheBlock(indexer, table, cols=None, where=None, distinct_on=None, order_by=None, limit=0, group_by=None, agg_on=None, index_on=None)

indexer, table, cols=None, where=None, distinct_on=None, order_by=None, limit=0, group_by=None, agg_on=None, index_on=None

Wraps up a set of query parameters for a specific entity, and provides cached access
to different types of "re-queries" via an associated Indexer.

to different types of “re-queries” via an associated Indexer.

The goal here is to help build/define entities as the possibly complex
transformations on the base schema that they are. For example, the Note primitive
(entity) incorporates details across ``files``, ``notes``, ``note_conversions``,
and ``note_conversion_matter`` tables (defined in a single endpoint by a
Composer), often needs to be selected in particular ways (via an Accessor), and
results stored for fast access later on (handled by an Indexer). This pipeline can
be daunting and requires too many moving parts to be handled explicitly
everywhere. CacheBlocks wrap up a set of query "preferences," exposing a simpler
interface for downstream access to entities. It still allows for low-level control
over re-grouping/indexing, raw hits to the actual DB, etc, but keeps things
tighter and well-behaved for the Indexer.

You can think of these as the Indexer's "fingers"; they're deployable mini-Indexes
that "send back" results to the class cache, which is "broadcast" to all other
instances for use when necessary.

(entity) incorporates details across 

 tables (defined in a single endpoint by a
everywhere. CacheBlocks wrap up a set of query “preferences,” exposing a simpler

You can think of these as the Indexer’s “fingers”; they’re deployable mini-Indexes
that “send back” results to the class cache, which is “broadcast” to all other

    cb = CacheBlock()

    # Set up cached queries with chained params or via call:

    cb.where(t.notes.c.name=="name").group_by(t.note_conversions.c.format)
    cb() # get results

    # - OR - # (use strings when known)

    cb.where(t.notes.c.name=="name").group_by('format')

    # - OR - # (use kwargs in the call; results returned right away)

    cb(
        where=(t.notes.c.name=="name"),
        group_by='format'
    )

cb = CacheBlock()

# Set up cached queries with chained params or via call:

cb.where(t.notes.c.name=="name").group_by(t.note_conversions.c.format)
cb() # get results

# - OR - # (use strings when known)

cb.where(t.notes.c.name=="name").group_by('format')

# - OR - # (use kwargs in the call; results returned right away)

cb(
    where=(t.notes.c.name=="name"),
    group_by='format'
)

CacheBlock.__init__(indexer, table, cols=None, where=None, distinct_on=None, order_by=None, limit=0, group_by=None, agg_on=None, index_on=None)

Provides restricted access to an underlying Accessor to enable more efficient, superficial
caching.

Cache clearing is to be handled by a wrapper class, like the Database.

Caching occurs at the class level, with indexes prefixed by table's origin Composer.
This means that cached selects/group-bys will be available regardless of the provided
Accessors so long as the same Composer is used under the hood.

Caching occurs at the class level, with indexes prefixed by table’s origin Composer.

Indexer.__init__(accessor, cache_select=True, cache_groupby=True)

Indexer.cached_query(table, cols=None, where=None, distinct_on=None, order_by=None, limit=0, group_by=None, agg_on=None, index_on=None)

table, cols=None, where=None, distinct_on=None, order_by=None, limit=0, group_by=None, agg_on=None, index_on=None

Like ``group_by``, but makes a full query to the Accessors table ``table_name`` and
caches the results. The processing performed by the GROUP BY is also cached.

 and

Update: ``cached_select`` and ``cached_group_by`` now unified by a single
``cached_query`` method. This allows better defined GROUP BY caches, that are
reactive to the full set of parameters returning the result set (and not just the
table, requiring a full query).

 method. This allows better defined GROUP BY caches, that are

Note: on cache keys
Cache keys are now fully stringified, as many objects are now allowed to be
native SQLAlchemy objects. Indexing these objects works, but doing so will
condition the cache on their memory addresses, which isn't what we want.
SQLAlchemy converts most join/column/table-like objects to reasonable strings,
which will look the same regardless of instance.

Context: this became a clear issue when passing in more
``order_by=<col>.desc()``. The ``desc()`` causes the index to store the column in
an instance-specific way, rather than an easily re-usable, canonical column
reference. Each time the CoreDatabase.files() was being called, for instance,
that @property would be re-evaluated, causing ``desc()`` to be re-initialized,
and thus look different to the cache. Stringifying everything prevents this
(although this could well be an indication that only a single ``cache_block``
should ever be returned be database properties).

condition the cache on their memory addresses, which isn’t what we want.

 causes the index to store the column in
that @property would be re-evaluated, causing 

 to be re-initialized,
(although this could well be an indication that only a single 

Note: on access locks
A double-checked locking scheme is employed before both of the stages (select
and manual group by), using the same lock. This resolves the common scenario
where many threads need to look up a query in the cache, experience a cache
miss, and try to do the work. This non-linearly explodes the total time to
wait in my experience, so doing this only when needed saves tons of time,
especially in high-congestion moments.

Indexer.group_by(rows, group_by, agg_on=None, index_on=None, return_index=False)

Post-query "group by"-like aggregation. Creates an index over a set of columns
(``group_by_cols``), and aggregates values from ``agg_cols`` under the groups.

Post-query “group by”-like aggregation. Creates an index over a set of columns
(

Rows can be dicts or mappings, and columns can be strings or SQLAlchemy columns.
To ensure the right columns are being used for the operation, it's best to pass in
mappings and use SQA columns if you aren't sure exactly how the keys look in your
results (dicts can have ambiguous keys across tables with the same columns and/or
different labeling schemes altogether).

To ensure the right columns are being used for the operation, it’s best to pass in
mappings and use SQA columns if you aren’t sure exactly how the keys look in your

TODO: add a flag that handles None's as distinct. That is, for the group_by
column(s) of interest, if rows in the provided query set have NULL values for
these columns, treat all such rows as their "own group" and return them alongside
the grouped/aggregated ones. This is behavior desired by something like
FTSManager.recreate(), which wants to bundle up conversions for blocks
(effectively grouping by blocks.name and link.id, aggregating on
block_conversions.format, then flattening). You could either do this, or as the
caller just make sure to first filter the result set before grouping (e.g.,
splitting the NULL-valued rows from those that are well-defined), and then
stitching the two sets back together afterward.

TODO: add a flag that handles None’s as distinct. That is, for the group_by
these columns, treat all such rows as their “own group” and return them alongside

group_by: can be a tuple of tuples of columns. Each inner tuple is a nested
"group by index" in the group by index

“group by index” in the group by index

Wrapper for state-changing database operations. Managers expose the formalized means of
interacting with an underlying database, like inserts and schema recreation.

The schema is centered around a connected group of tables (via foreign keys). Thus,
most operations need to be coordinated across tables. A few common operations are
wrapped up in this class to then also be mirrored for the FTS counterparts.

Note: Common on insert behavior
- Tables with unique constraints have been equipped with ``sqlite_on_conflict_unique``
  flags, enabling conflicting bulk inserts to replace conflicting rows gracefully. No
  need to worry about explicitly handling upserts.
- The bulk insert via conn.execute(<insert>,<row_list>) automatically ignores
  irrelevant column names within provided record dicts, whereas explicit .values() calls
  for non-bulk inserts will throw errors if not aligned perfectly. We want to keep these
  bulk calls AND allow update/replacements on conflict; the setting above allows this
  bulk usage to continue as is.

Tables with unique constraints have been equipped with ``sqlite_on_conflict_unique``
flags, enabling conflicting bulk inserts to replace conflicting rows gracefully. No
need to worry about explicitly handling upserts.

The bulk insert via conn.execute(<insert>,<row_list>) automatically ignores
irrelevant column names within provided record dicts, whereas explicit .values() calls
for non-bulk inserts will throw errors if not aligned perfectly. We want to keep these
bulk calls AND allow update/replacements on conflict; the setting above allows this
bulk usage to continue as is.

Note: Options for insert/update model
1. Collect all possible update objects, and use a SQLite bulk insert that only updates
   if needed (based on modified time, for example). This sounds on the surface like
   the right approach since we defer as much to bulk SQL logic, but it's far and away
   the worse option b/c prepping all file/note/etc objects is too expensive to
   ultimately throw away when finding out an update isn't needed. For example, if we
   wanted to perform note updates _inside_ the SQL call (like an ``INSERT .. UPDATE ..
   IF``, as opposed to determining the objects to insert _outside_ of the SQL call),
   you would need to bring each of the note's HTML prior to the insert check. There's
   no 2-stage processing here where you can check if the note needs to be converted
   b/c it's out of date, and only then perform the computation.
2. Instantiate objects sequentially, each time checking with the DB to see if full
   processing is needed. This makes much more sense when the full processing is very
   expensive, as it is with Note conversion. This would iterate through available notes,
   perform a ``SELECT`` on the target table to see if the note needs updating, and if so
   perform the remaining computation. Those objects then get added to a "update object
   list" to be inserted in bulk, but you make sequential ``SELECT`` checks before that.

   The one extra optimization you could maybe make here is doing a full SELECT on the
   target table and bring all rows into memory before iterating through the objects.
   This would likely make it faster than whatever SQLAlchemy overhead there may be. It
   also might just be outright required given Connection objects aren't thread-safe;
   we can get away with single thread global SELECT, threaded checking during object
   build, then single thread bulk INSERT. (**Note**: this is what the method does).

Collect all possible update objects, and use a SQLite bulk insert that only updates
if needed (based on modified time, for example). This sounds on the surface like
the right approach since we defer as much to bulk SQL logic, but it's far and away
the worse option b/c prepping all file/note/etc objects is too expensive to
ultimately throw away when finding out an update isn't needed. For example, if we
wanted to perform note updates _inside_ the SQL call (like an ``INSERT .. UPDATE ..
IF``, as opposed to determining the objects to insert _outside_ of the SQL call),
you would need to bring each of the note's HTML prior to the insert check. There's
no 2-stage processing here where you can check if the note needs to be converted
b/c it's out of date, and only then perform the computation.

the right approach since we defer as much to bulk SQL logic, but it’s far and away
ultimately throw away when finding out an update isn’t needed. For example, if we
wanted to perform note updates _inside_ the SQL call (like an 

, as opposed to determining the objects to insert _outside_ of the SQL call),
you would need to bring each of the note’s HTML prior to the insert check. There’s
b/c it’s out of date, and only then perform the computation.

Instantiate objects sequentially, each time checking with the DB to see if full
processing is needed. This makes much more sense when the full processing is very
expensive, as it is with Note conversion. This would iterate through available notes,
perform a ``SELECT`` on the target table to see if the note needs updating, and if so
perform the remaining computation. Those objects then get added to a "update object
list" to be inserted in bulk, but you make sequential ``SELECT`` checks before that.

The one extra optimization you could maybe make here is doing a full SELECT on the
target table and bring all rows into memory before iterating through the objects.
This would likely make it faster than whatever SQLAlchemy overhead there may be. It
also might just be outright required given Connection objects aren't thread-safe;
we can get away with single thread global SELECT, threaded checking during object
build, then single thread bulk INSERT. (**Note**: this is what the method does).

perform a 

 on the target table to see if the note needs updating, and if so
perform the remaining computation. Those objects then get added to a “update object
list” to be inserted in bulk, but you make sequential 

also might just be outright required given Connection objects aren’t thread-safe;
build, then single thread bulk INSERT. (

Bases: :py:class:`~co3.manager.Manager`, :py:class:`~typing.Generic`

Core schema table manager. Exposes common operations and facilitates joint operations
needed for highly connected schemas.

In particular, Managers expose insertion abstractions that take table-indexed groups
of rows and bundle them under a single transaction. This is important for table groups
with foreign keys and cascading deletions: inserts need to be coordinated. Note that
actually collecting the inserts that need to take place is outside the scope of the
Manager (see the Collector). We do, however, implement a ``sync`` operation that can
saturates a router with events (dynamically) and sweeps up inserts on session basis
from an attached collector.

Manager (see the Collector). We do, however, implement a 

 operation that can

The insert lock is a _reentrant lock_, meaning the same thread can acquire the
lock again with out deadlocking (simplifying across methods of this class that
need it).

, meaning the same thread can acquire the
need it).

Ideally this remains open, as we can't necessarily rely on a SQLAlchemy metadata
object for all kinds of SQLDatabases (would depend on the backend, for instance).

Ideally this remains open, as we can’t necessarily rely on a SQLAlchemy metadata

Haven't quite nailed down how backend instances should be determined; something
like SQLAlchemySQLManager doesn't seem great. Nevertheless, this method likely
cannot be generalized at the "SQL" (general) level.

Haven’t quite nailed down how backend instances should be determined; something
like SQLAlchemySQLManager doesn’t seem great. Nevertheless, this method likely
cannot be generalized at the “SQL” (general) level.

SQLManager.sync(limit=0, chunk_time=0, dry=False, chunk_size_cap=1000, print_report=True)

limit=0, chunk_time=0, dry=False, chunk_size_cap=1000, print_report=True

Could be dangerous if going from fast file processing to note processing.
Imagine estimating 1000 iter/sec, then transferring that to the next batch
when it's more like 0.2 iter/sec. We would lose any chunking. (Luckily, in
practice, turns out that notes are almost always processed before the huge set
of nested files lurking and we don't experience this issue.)

when it’s more like 0.2 iter/sec. We would lose any chunking. (Luckily, in
of nested files lurking and we don’t experience this issue.)

Used to house useful objects for storage schemas (e.g., SQLAlchemy table definitions).
Provides a general interface for mapping from CO4 class names to storage structures for
auto-collection and composition.

mapper = Mapper[sa.Table]()

mapper.attach(
    Type,
    attr_comp=TypeTable,
    coll_comp=CollateTable,
    coll_groups={
        'name': NameConversions
    }
)

- Overruled design decision: Mappers were previously designed to map from a specific
  CO3 hierarchy to a specific Schema. The intention was to allow only related types to
  be attached to a single schema, at least under a particular Mapper. The type
  restriction has since been removed, however, as it isn't particularly well-founded.
  During ``collect()``, a particular instance collects data from both its attributes and
  its collation actions. It then repeats the same upward for parent types (part of the
  same type hierarchy), and down to components (often not part of the same type
  hierarchy). As such, to fully collect from a type, the Mapper needs to leave
  registration open to various types, not just those part of the same hierarchy.

Overruled design decision: Mappers were previously designed to map from a specific
CO3 hierarchy to a specific Schema. The intention was to allow only related types to
be attached to a single schema, at least under a particular Mapper. The type
restriction has since been removed, however, as it isn't particularly well-founded.
During ``collect()``, a particular instance collects data from both its attributes and
its collation actions. It then repeats the same upward for parent types (part of the
same type hierarchy), and down to components (often not part of the same type
hierarchy). As such, to fully collect from a type, the Mapper needs to leave
registration open to various types, not just those part of the same hierarchy.

restriction has since been removed, however, as it isn’t particularly well-founded.
During 

, a particular instance collects data from both its attributes and

ComposableMapper(schema, attr_compose_map=None, coll_compose_map=None)

Bases: :py:class:`~co3.mapper.Mapper`, :py:class:`~typing.Generic`

Heavily debating between multiple possible design approaches here. The main
purpose of this subtype is make clear the need for additional compositional
mapping details, namely functions that can produce pairwise join conditions for
both the attribute tree (vertical traversal) and the collation components
(horizontal traversal). Here's a few remarks:

- I want the necessary maps to provided/stored *outside* of ``compose`` calls to
  reduce overhead for downstream callers. It's awkward to have think about the
  exact attr-to-attr associations each time you want a type's associated
  composition, especially when they don't change under the same Mapper (i.e.,
  if you have the Mapper reference, the compositional associations should be
  implicit).
- The barebones spec here appears to be two pairwise "composer" maps: one for
  attribute comps, and one for collation comps. For now I think this makes sense
  as additional init params, but there may later be reason to wrap this up a bit
  more.
- Considering the full deprecation for the Composer type, or whether this could be
  the place where it serves some purpose. Aesthetically, there's symmetry with the
  ``collect`` and Collector method-type pairing, but that isn't a good enough reason
  to justify a separate type here. The difference is that Collector instances
  actually store type references, whereas the considered Composer type would
  effectively just be a convenient collection of utility functions. Still possibly
  useful, but not as clearly justifiable.
- If a separate Composer type were to be justified here, it would serve as a
  "reusable connective tissue" for possibly many Mappers with the same kinds of
  edge-wise relationships. Can think of it like this:

  * Schemas collect up "nodes" (Components). These are explicit storage structures
    in a DB, and can include some explicit attribute connections (foreign keys),
    although those are connections made on the DB side.
  * Mappers provide an exoskeleton for a Schema's nodes. It structures Components into
    attributes and collation types, and additionally ties them to external CO3
    types. The handy analogy here has been that attribute comps connect
    *vertically* (in a tree like fashion; point up for parents and down for
    children), and collation comps point *horizontally* (or perhaps more aptly,
    *outward*; at each node in the attribute tree, you have a "circle" of
    collation comps that can point to it, and are not involved as formal tree
    nodes. Can maybe think of these like "ornaments" or bulbs or orbitals).
  * While the Mappers may provide the "bones," there's no way to communicate
    *across* them. While I might know that one attribute is the "parent" of
    another, I don't know *why* that relationship is there. A Composer, or the
    composer details to be provided to this class, serve as the "nerves" to be
    paired with the bone, actually establishing a line of communication. More
    specifically, the nerves here are attribute-based mappings between pairs of
    Components, i.e., (generalized) join conditions.

- Note that, by the above logic, we should then want/need a type to manage the
  functions provided to ``attach_many``. These functions help automatically
  characterize the shape of the type skeleton in the same way the proposed
  Composer wrapper would. In fact, the barebones presentation here is really just
  the same two function signatures as are expected by that method. The above
  analogy simply made me ask why the "bones" wouldn't be reusable if the "nerves"
  were going to be. So we should perhaps coordinate a decision on this front; if
  one goes, the other must as well. This may also help me keep it simpler for the
  time being.
- One other aspect of a dedicated Composer type (and by the above point, a
  hypothetical type to aid in ``attach_many`` specification) could have some sort of
  "auto" feature about it. With a clear enough "discovery system," we could
  encourage certain kinds of Schemas and components are named and structured. Such
  an auto-composer could "scan" all components in a provided Schema and attempt to
  find common attributes across tables that are unlinked (i.e., the reused
  column names implicit across types in the attribute hierarchy; e.g., File.name
  -> Note.name), as well as explicit connections which may suggest collation
  attachment (e.g., ``note_conversions.name --FK-> Note.name``). This, of course,
  could always be overridden with manual specification, but being aware of some
  automatic discovery structures could help constrain schema definitions to be
  more in-line with the CO3 operational model. That all being said, this is a
  large amount of complexity and should likely be avoided until necessary.

(horizontal traversal). Here’s a few remarks:

I want the necessary maps to provided/stored *outside* of ``compose`` calls to
reduce overhead for downstream callers. It's awkward to have think about the
exact attr-to-attr associations each time you want a type's associated
composition, especially when they don't change under the same Mapper (i.e.,
if you have the Mapper reference, the compositional associations should be
implicit).

 calls to
reduce overhead for downstream callers. It’s awkward to have think about the
exact attr-to-attr associations each time you want a type’s associated
composition, especially when they don’t change under the same Mapper (i.e.,

The barebones spec here appears to be two pairwise "composer" maps: one for
attribute comps, and one for collation comps. For now I think this makes sense
as additional init params, but there may later be reason to wrap this up a bit
more.

The barebones spec here appears to be two pairwise “composer” maps: one for
more.

Considering the full deprecation for the Composer type, or whether this could be
the place where it serves some purpose. Aesthetically, there's symmetry with the
``collect`` and Collector method-type pairing, but that isn't a good enough reason
to justify a separate type here. The difference is that Collector instances
actually store type references, whereas the considered Composer type would
effectively just be a convenient collection of utility functions. Still possibly
useful, but not as clearly justifiable.

the place where it serves some purpose. Aesthetically, there’s symmetry with the

 and Collector method-type pairing, but that isn’t a good enough reason

If a separate Composer type were to be justified here, it would serve as a
"reusable connective tissue" for possibly many Mappers with the same kinds of
edge-wise relationships. Can think of it like this:

* Schemas collect up "nodes" (Components). These are explicit storage structures
  in a DB, and can include some explicit attribute connections (foreign keys),
  although those are connections made on the DB side.
* Mappers provide an exoskeleton for a Schema's nodes. It structures Components into
  attributes and collation types, and additionally ties them to external CO3
  types. The handy analogy here has been that attribute comps connect
  *vertically* (in a tree like fashion; point up for parents and down for
  children), and collation comps point *horizontally* (or perhaps more aptly,
  *outward*; at each node in the attribute tree, you have a "circle" of
  collation comps that can point to it, and are not involved as formal tree
  nodes. Can maybe think of these like "ornaments" or bulbs or orbitals).
* While the Mappers may provide the "bones," there's no way to communicate
  *across* them. While I might know that one attribute is the "parent" of
  another, I don't know *why* that relationship is there. A Composer, or the
  composer details to be provided to this class, serve as the "nerves" to be
  paired with the bone, actually establishing a line of communication. More
  specifically, the nerves here are attribute-based mappings between pairs of
  Components, i.e., (generalized) join conditions.

“reusable connective tissue” for possibly many Mappers with the same kinds of

Schemas collect up "nodes" (Components). These are explicit storage structures
in a DB, and can include some explicit attribute connections (foreign keys),
although those are connections made on the DB side.

Schemas collect up “nodes” (Components). These are explicit storage structures

Mappers provide an exoskeleton for a Schema's nodes. It structures Components into
attributes and collation types, and additionally ties them to external CO3
types. The handy analogy here has been that attribute comps connect
*vertically* (in a tree like fashion; point up for parents and down for
children), and collation comps point *horizontally* (or perhaps more aptly,
*outward*; at each node in the attribute tree, you have a "circle" of
collation comps that can point to it, and are not involved as formal tree
nodes. Can maybe think of these like "ornaments" or bulbs or orbitals).

Mappers provide an exoskeleton for a Schema’s nodes. It structures Components into

 (in a tree like fashion; point up for parents and down for
children), and collation comps point 

; at each node in the attribute tree, you have a “circle” of
nodes. Can maybe think of these like “ornaments” or bulbs or orbitals).

While the Mappers may provide the "bones," there's no way to communicate
*across* them. While I might know that one attribute is the "parent" of
another, I don't know *why* that relationship is there. A Composer, or the
composer details to be provided to this class, serve as the "nerves" to be
paired with the bone, actually establishing a line of communication. More
specifically, the nerves here are attribute-based mappings between pairs of
Components, i.e., (generalized) join conditions.

While the Mappers may provide the “bones,” there’s no way to communicate

 them. While I might know that one attribute is the “parent” of
another, I don’t know 

 that relationship is there. A Composer, or the
composer details to be provided to this class, serve as the “nerves” to be

Note that, by the above logic, we should then want/need a type to manage the
functions provided to ``attach_many``. These functions help automatically
characterize the shape of the type skeleton in the same way the proposed
Composer wrapper would. In fact, the barebones presentation here is really just
the same two function signatures as are expected by that method. The above
analogy simply made me ask why the "bones" wouldn't be reusable if the "nerves"
were going to be. So we should perhaps coordinate a decision on this front; if
one goes, the other must as well. This may also help me keep it simpler for the
time being.

functions provided to 

. These functions help automatically
analogy simply made me ask why the “bones” wouldn’t be reusable if the “nerves”

One other aspect of a dedicated Composer type (and by the above point, a
hypothetical type to aid in ``attach_many`` specification) could have some sort of
"auto" feature about it. With a clear enough "discovery system," we could
encourage certain kinds of Schemas and components are named and structured. Such
an auto-composer could "scan" all components in a provided Schema and attempt to
find common attributes across tables that are unlinked (i.e., the reused
column names implicit across types in the attribute hierarchy; e.g., File.name
-> Note.name), as well as explicit connections which may suggest collation
attachment (e.g., ``note_conversions.name --FK-> Note.name``). This, of course,
could always be overridden with manual specification, but being aware of some
automatic discovery structures could help constrain schema definitions to be
more in-line with the CO3 operational model. That all being said, this is a
large amount of complexity and should likely be avoided until necessary.

hypothetical type to aid in 

 specification) could have some sort of
“auto” feature about it. With a clear enough “discovery system,” we could
an auto-composer could “scan” all components in a provided Schema and attempt to
attachment (e.g., 

). This, of course,

ComposableMapper.__init__(schema, attr_compose_map=None, coll_compose_map=None)

schema (Schema[TypeVar(C, bound= ComposableComponent)]) -- Schema object holding the set of components eligible as attachment
targets for registered CO3 types

Schema object holding the set of components eligible as attachment

ComposableMapper.compose(co3_ref, action_groups=None, *compose_args, **compose_kwargs)

Compose tables up the type hierarchy, and across through action groups to
collation components.

Comparing to ORM, this method would likely also still be needed, since it may
not be explicitly clear how some JOINs should be handled up the inheritance
chain (for components / sa.Relationships, it's a little easier).

chain (for components / sa.Relationships, it’s a little easier).

Mapper base class for housing schema components and managing relationships between CO3
types and storage components (of type C).

Attaching CO3 types to database Components from within a single schema

Facilitating collection of Component-related insertion data from instances of
attached CO3 types

Additionally, the Mapper manages its own Collector and Composer instances. The
Collector receives the inserts from ``.collect()`` calls, and will subsequently be
"dropped off" at an appropriate Database's Manager to actually perform the requested
inserts (hence why we tie Mappers to Schemas one-to-one).

Collector receives the inserts from 

 calls, and will subsequently be
“dropped off” at an appropriate Database’s Manager to actually perform the requested

- Consider pushing this into a Mapper factory; on init, could check if provided
  Schema wraps up composable Components or not

Dev note
the Composer needs reconsideration, or at least its positioning directly in this
class. It may be more appropriate to have at the Schema level, or even just
dissolved altogether if arbitrary named Components can be attached to schemas.

Consider pushing this into a Mapper factory; on init, could check if provided
Schema wraps up composable Components or not

schema (Schema[TypeVar(C, bound= Component)]) -- Schema object holding the set of components eligible as attachment

Mapper.attach(type_ref, attr_comp, coll_comp=None, coll_groups=None)

type_ref (type[CO3]) -- CO3 subtype to map to provided storage components

attr_comp (Union[str, TypeVar(C, bound= Component)]) -- storage component for provided type's canonical attributes

coll_comp (Union[str, TypeVar(C, bound= Component), None]) -- storage component for provided type's default/unnamed collation
targets

storage component for provided type’s default/unnamed collation
targets

coll_groups (dict[str | None, Union[str, TypeVar(C, bound= Component)]] | None) -- storage components for named collation groups; dict mapping group
names to components

storage components for named collation groups; dict mapping group

Mapper.attach_many(type_list, attr_name_map, coll_name_map=None)

Auto-register a set of types to the Mapper's attached Schema. Associations are
made from types to both attribute and collation component names, through
``attr_name_map`` and ``coll_name_map``, respectively. Collation targets are inferred
through the registered groups in each type.

Auto-register a set of types to the Mapper’s attached Schema. Associations are

, respectively. Collation targets are inferred

attr_name_map (Callable[[type[CO3]], Union[str, TypeVar(C, bound= Component)]]) -- function mapping from types/classes to attribute component names
in the attached Mapper Schema

function mapping from types/classes to attribute component names

coll_name_map (Optional[Callable[[type[CO3], str], Union[str, TypeVar(C, bound= Component)]]]) -- function mapping from types/classes & action groups to
collation component names in the attached Mapper Schema. None
is passed as the action group to retrieve the default
collection target.

function mapping from types/classes & action groups to
collation component names in the attached Mapper Schema. 

Stages inserts up the inheritance chain, and down through components.

Even with ORM, a method like this would be needed to trace up parent tables and
how inserts should be handled for inheritance. ORM would make component
inserts a little easier perhaps, since they can be attached as attributes to
constructed row objects and a sa.Relationship will handle the rest. Granted,
we don't do a whole lot more here: we just call ``collect`` over those
components, adding them to the collector session all the same.

we don’t do a whole lot more here: we just call 

 over those

Returns: dict with keys and values relevant for associated SQLite tables

Collection of related storage components, often representing the data structure of an
entire database. Some databases support multiple schemas, however. In general, a Schema
can wrap up an associated subset of components within a single database, so long as
``Manager.recreate()`` supports creating components in separate calls (even if the
associated database doesn't explicitly support multiple schemas).

 supports creating components in separate calls (even if the
associated database doesn’t explicitly support multiple schemas).

Provide target contexts for connected CO3 type systems within Mappers

with particular emphasis on the latter. Mappers associate exactly one CO3 type hierarchy
with exactly one Schema. This is an intentional point of simplification in the CO3
operational model.

Bases: :py:class:`~co3.schema.Schema`, :py:class:`~typing.Generic`

# get SA engine, creating folder hierarchy to provided DB path
engine = db.get_engine(<path>)

# execute a single SA statement, returns a CursorResult
select_results = db.sa_execute(engine, sa.select(<table>))

# convert raw results to dictionaries, keys corresponding to col names
select_dicts = db.named_results(<table>, select_results)

# use table defaults and cols to create compliant insert
insert_dicts = [ db.prepare_insert(<table>, sd) for sd in select_dicts ]

# perform a bulk insert
with engine.connect() as connection:
    connection.execute(
        sa.insert(<table>),
        insert_dicts
    )

create_fts5(engine, table, columns=None, populate=False, inserts=None, reset_fts=False, tokenizer='unicode61')

engine, table, columns=None, populate=False, inserts=None, reset_fts=False, tokenizer='unicode61'

Create and optionally populate an FTS5 table in SQLite. Can be used directly for
existing tables in the same database. It can also be used for composite tables (i.e.,
those created from JOINs) or really any other data by providing explicit inserts and
column names to use during population.

table (Table | str) -- either SQLAlchemy table instance, or table name string

columns -- list of SQLAlchemy table columns to insert into virtual table. These
columns must be present in the provided table if not manually specifying
inserts (since the table must be queried automatically)

list of SQLAlchemy table columns to insert into virtual table. These

create_vss0(engine, table, columns=None, populate=False, inserts=None, reset=False, embedding_size=384)

engine, table, columns=None, populate=False, inserts=None, reset=False, embedding_size=384

Prefer this when using FKEYs; need to really justify *not* having a CASCADE deletion
enabled

Note the implications of this for results from compound tables containing the same
column names: only the last column name will be indexed.

co3.Accessor

co3.Collector

co3.CO3

Conversion & DB insertion base class

co3.Database

Generic Database definition

co3.Indexer

Indexer class

co3.Manager

Management wrapper class for table groupings.

co3.Mapper

Mapper base class for housing schema components and managing relationships between CO3 types and storage components (of type C).

co3.Component

co3.Schema

co3.Engine

Engine base class.

`co3` is a package for file conversion and associated database operations. The `CO3` base class
provides a standard interface for performing conversions, preparing inserts, and
interacting with database schemas that mirror the class hierarchy.

 is a package for file conversion and associated database operations. The 

**Goal**: interact with a storage medium (database, pickled structure, VSS framework) with
a known schema.

: interact with a storage medium (database, pickled structure, VSS framework) with

**Composer** to compose common access points (e.g., JOINed tables)

**Manager** to manage storage state (e.g., supported inserts, database syncs)

 to manage storage state (e.g., supported inserts, database syncs)

**CO3** is an abstract base class that makes it easy to integrate this model with object
hierarchies that mirror a storage schema.

 is an abstract base class that makes it easy to integrate this model with object

There are a few pillars of the CO3 model that meaningfully group up functionality:

Database: generic to a Component type, provides basic connection to a database at a
specific address/location. The explicit Component type makes it easy to hook into
appropriately typed functional objects:

Manager: generic to a Component and Database type, provides a supported set of
state-modifying operations to a constituent database

Accessor: generic to a Component and Database type, provides a supported set of
state inspection operations on a constituent database

Mapper: generic to a Component, serves as the fundamental connective component between
types in the data representation hierarchy (CO3 subclasses) and database Components.

The primary driver of this package's documentation is Sphinx's `autodoc` extension,
using the [Furo theme][1].

`sphinx-apidoc` generates package-based documentation to the `_autoref/` directory,
with navigation available under "Autoref" in the sidebar.

Markdown-based documentation files are manually written under the `reference/`
directory, showing up under "Contents" in the sidebar.

Markdown-based documentation files are manually written under the 

`_autoref/` is the target for module-based RST files written by `autodoc`

`reference/`: houses all manually written documentation (totally separate from
auto-generated package docs)

: houses all manually written documentation (totally separate from

`index.md`: documentation index, setups up a persistent sidebar across all other pages

: documentation index, setups up a persistent sidebar across all other pages

For manually written documentation under `reference/`, topics are nested as needed. Within
a nested directory `reference/<topic>`, an `index.md` should created with content like:

# <Topic>

\`\`\`{toctree}
:hidden:

sub-topic-1.rst
sub-topic-2.rst
...
\`\`\`

This will add the nested directory to the sidebar navigation, using the name set under the
top-level header. See [Markdown syntax][#markdown-syntax] for more details on the syntax.

Sphinx's `autodoc` extension allows automatic generation of documents according to
(Python) subpackage structure and available docstrings. A few notes here:

 extension allows automatic generation of documents according to

In the `conf.py` file, autodoc is enabled by adding `"sphinx.ext.autodoc"` to
the extensions list. `"sphinx.ext.viewcode"` can also be added to provide
links to source code.

Documents are actually generated by calling the `sphinx-apidoc` CLI command. The
current Makefile uses the following call:

This writes the automatically generated docs for modules in the package at the
local directory `localsys/` to the `docs/sphinx/_autoref` directory. These are
reStructuredText files by default.

`--module-first` places the module-level descriptions at the top of the module page.
By default, this is placed at the bottom (oddly), and can be obscured by large lists
of subpackages if this flag isn't provided.

 places the module-level descriptions at the top of the module page.

See available `sphinx-apidoc` options [here][2], as well as more advanced config
[here][3].

The `myst_parser` extension enables Markdown (or something close to it) to be used when
writing documentation files. The Sphinx directives can be difficult to track, and
they change slightly under the MyST Markdown syntax. The following are a few common
blocks:

 extension enables Markdown (or something close to it) to be used when

**Page hierarchies**: the following will generate link hierarchy according to the provided
pages:

: the following will generate link hierarchy according to the provided

:maxdepth: <n>
:caption: <caption>
:hidden:

example-file-1
example-file-2
example-dir/index
...
\`\`\`

`:hidden:` if provided, links will only show in the sidebar (hidden on the page)

 if provided, links will only show in the sidebar (hidden on the page)

Constituent files: listed files will be rendered as a link directly. If a listed file
has a `{toctree}` directive, this tree will be rendered in place of the page's link as a
dropdown. The dropdown will be named according to the file's top-level heading, and
clicking directly on the dropdown header will show that page's content. Files found in
the tree will be placed as links under the dropdown, recursively subject to same rules
described here.

 directive, this tree will be rendered in place of the page’s link as a

dropdown. The dropdown will be named according to the file’s top-level heading, and

clicking directly on the dropdown header will show that page’s content. Files found in

**Include files**: the following will include file content
pages:

'''
Accessor

'''
import inspect
from pathlib import Path
from collections import defaultdict
from abc import ABCMeta, abstractmethod

import sqlalchemy as sa

from co3.component import Component

class Accessor[C: Component](metaclass=ABCMeta):
    '''
    Access wrapper class for complex queries and easy integration with Composer tables.
    Implements high-level access to things like common constrained SELECT queries.

    Parameters:
        engine: SQLAlchemy engine to use for queries. Engine is initialized dynamically as
                a property (based on the config) if not provided
    '''
    @abstractmethod
    def raw_select(
        self,
        text: str,
    ):
        raise NotImplementedError

    def select(
        self,
        component: C,
        *args,
        **kwargs
    ):

from co3 import util
from co3.accessor import Accessor

class FTSAccessor(Accessor):
    def search(
        self,
        table_name  : str,
        select_cols : str  | list | None = '*',
        search_cols : str  | None = None,
        q           : str  | None = None,
        colq        : str  | None = None,
        snip_col    : int  | None = 0,
        hl_col      : int  | None = 0,
        limit       : int  | None = 100,
        snip        : int  | None = 64,
        tokenizer   : str  | None = 'unicode61',
        group_by    : str  | None = None,
        agg_cols    : list | None = None,
        wherein_dict: dict | None = None,
        unique_on   : dict | None = None,
    ):
        '''
        Execute a search query against an indexed FTS table for specific primitives. This
        method is mostly a generic FTS handler, capable of handling queries to any available
        FTS table with a matching naming scheme (``fts_<type>_<tokenizer>``). The current
        intention is support all tokenizers, for file, note, block, and link primitives.

        Search results include all FTS table columns, as well as SQLite-supported ``snippet``s
        and ``highlight``s for matches. Matches are filtered and ordered by SQLite's
        ``MATCH``-based score for the text & column queries. Results are (a list of) fully
        expanded dictionaries housing column-value pairs.

        Note:
            GROUP BY cannot be paired with SQLITE FTS extensions; thus, we perform manual
            group checks on the result set in Python before response

        Analysis:
            The returned JSON structure has been (loosely) optimized for speed on the client
            side. Fully forming individual dictionary based responses saves time in
            Javascript, as the JSON parser is expected to be able to create the objects
            faster than post-hoc construction in JS. This return structure was compared
            against returning an array of arrays (all ordered in the same fashion), along with
            a column list to be associated with each of the result values. While this saves
            some size on the payload (the same column names don't have to be transmitted for
            each result), the size of the returned content massively outweighs the
            predominantly short column names. The only way this structure would be viable is
            if a significant amount was saved on transfer compared to the slow down in JS
            object construction; this is (almost) never the case.

        Parameters:
            table_name  : name of FTS table to search
            search_cols : space separated string of columns to use for primary queries
            q           : search query
            colq        : column constraint string; must conform to SQLite standards (e.g.,
                          ``<col>:<text>``
            snip_col    : table column to use for snippets (default: 1; source content column)
            hl_col      : table column to use for highlights (default: 2; format column, applied
                          to HTML targets)
            limit       : maximum number of results to return in the SQL query
            snip        : snippet length (max: 64)
            tokenizer   : tokenizer to use (assumes relevant FTS table has been built)
            ...
            wherein_dict: (col-name, value-list) pairs to match result set against, via
                          WHERE ... IN clauses

        Returns:
            Dictionary with search results (list of column indexed dictionaries) and relevant
            metadata.
        '''
        search_q = ''

        if type(select_cols) is list:
            select_cols = ', '.join(select_cols)

        # construct main search query
        if search_cols and q:
            search_q = f'{{{search_cols}}} : {q}'

        # add auxiliary search constraints
        if colq:
            search_q += f' {colq}'

        search_q = search_q.strip()

        hl_start = '<b><mark>'
        hl_end   = '</mark></b>'

        fts_table_name = f'{table_name}_fts_{tokenizer}'
        
        sql = f'''
        SELECT
            {select_cols},
            snippet({fts_table_name}, {snip_col}, '{hl_start}', '{hl_end}', '...', {snip}) AS snippet,
            highlight({fts_table_name}, {hl_col}, '{hl_start}', '{hl_end}') AS highlight 
        FROM {fts_table_name}
        '''
        
        where_clauses = []
        if search_q:
            where_clauses.append(f"{fts_table_name} MATCH '{search_q}'\n")

        if wherein_dict:
            for col, vals in wherein_dict.items():
                where_clauses.append(f'{col} IN {tuple(vals)}\n')

        if where_clauses:
            where_str = " AND ".join(where_clauses)
            sql += f'WHERE {where_str}'

        sql += f'ORDER BY rank LIMIT {limit};'

        row_dicts, cols = self.raw_select(sql, include_cols=True)

        if group_by is None:
            return row_dicts, cols

        if agg_cols is None:
            agg_cols = []

        # "group by" block ID and wrangle the links into a list
        # note we can't perform native GROUP BYs with FTS results
        group_by_idx = {}
        for row in row_dicts:
            group_by_attr = row.get(group_by)

            # add new entries
            for agg_col in agg_cols:
                row[f'{agg_col}_agg'] = set()

            if group_by_attr is None:
                continue

            if group_by_attr not in group_by_idx:
                group_by_idx[group_by_attr] = row

                if agg_col in row:
                    group_by_idx[group_by_attr][f'{agg_col}_agg'].add(row[agg_col])

        return {
            'results'     : group_by_idx,
            'columns'     : cols,
            'num_results' : len(row_dicts),
        }

'''
Design proposal: variable backends

    class SQLAlchemyAccessor(RelationalAccessor): # may also inherit from a dedicated interface parent
        def select(...):
            ...
    
    class SQLiteAccessor(RelationalAccessor): 
            ...
    
    class SQLAccessor(RelationalAccessor): 
        backends = {
            'sqlalchemy': SQLAlchemyAccessor,
            'sqlite':     SQLteAccessor,
        }
    
        def __init__(self, backend: str):
            self.backend = self.backends.get(backend)
    
            return self.backend.select(...)

'''

from collections.abc import Iterable
from functools import cache

from co3.engines import SQLEngine
from co3.components import Relation, SQLTable

class RelationalAccessor[R: Relation](Accessor[R]):
        self, 
        text: str
    ):
        #connection.exec

        self,
        relation: R,
        attributes   = None,
        where        = None,
        distinct_on  = None,
        order_by     = None,
        limit        = 0,
        mappings     : bool = False,
        include_cols : bool = False,
    ):

    def select_one(
        self,
        relation     : R,
        attributes          = None,
        where               = None,
    ):
        res = self.select(
            relation, attributes, where, mappings, include_cols, limit=1
        )

        if include_cols and len(res[0]) > 0:
            return res[0][0], res[1]

        if len(res) > 0:
            return res[0]

        return None

class SQLAccessor(RelationalAccessor[SQLTable]):
        self,
        sql,
        bind_params=None,
        mappings=False,
        include_cols=False,
    ):
        res = SQLEngine.execute(
            connection,
            sql,
            bind_params=bind_params,
            include_cols=include_cols
        )

        if mappings:
            return res.mappings().all()
        else:
            return self.result_dicts(res)

        self,
        component:   SQLTable,
        columns      = None,
        limit        = 0,
        mappings     = False,
        include_cols = False,
    ): # -> list[dict|sa.Mapping]: (double check the Mapping types)
        '''
        Perform a SELECT query against the provided table-like object (see
        ``check_table()``).

        Deprecated: String aliases
            String aliases for tables are no longer supported. This method no longer checks
            against any specific schema table-maps or Composers. Instead, this should be
            done outside the Accessor.

            group_by: list of columns to group by; for now serves as a proxy for DISTINCT
                      (no aggregation methods accepted)
            order_by: column to order results by (can use <col>.desc() to order
                      by descending)

        Returns:
            Statement results, either as a list of 1) SQLAlchemy Mappings, or 2) converted
            dictionaries
        '''
        if where is None:
            where = sa.true()

        table = component.obj
        statement = sa.select(table).where(where)
        if columns is not None:
            statement = sa.select(*columns).select_from(table).where(where)

        if distinct_on is not None:
            statement = statement.group_by(distinct_on)

        if order_by is not None:
            statement = statement.order_by(order_by)

        if limit > 0:
            statement = statement.limit(limit)

        res = SQLEngine.execute(connection, statement, include_cols=include_cols)

        else:

    @staticmethod
    def result_dicts(results, query_cols=None):
        '''
        Parse SQLAlchemy results into Python dicts. Leverages mappings to associate full
        column name context.

        If ``query_cols`` is provided, their implicit names will be used for the keys of the
        returned dictionaries. This information is not available under CursorResults and thus
        must be provided separately. This will yield results like the following:

        [..., {'table1.col':<value>, 'table2.col':<value>, ...}, ...]

        Instead of the automatic mapping names:

        [..., {'col':<value>, 'col_1':<value>, ...}, ...]

        which can make accessing certain results a little more intuitive. 
        '''
        result_mappings = results.mappings().all()

        if query_cols:
            return [
                { str(c):r[c] for c in query_cols }
                for r in result_mappings
            ]

        return [dict(r) for r in result_mappings]

import pickle
import logging
import time

#from sentence_transformers import SentenceTransformer, util

logger = logging.getLogger(__name__)

class VSSAccessor(Accessor):
    def __init__(self, cache_path):
        super().__init__()

        self._model      = None
        self._embeddings = None

        self._embedding_size = 384
        self.embedding_path = Path(cache_path, 'embeddings.pkl')

    def write_embeddings(self, embedding_dict):
        self.embedding_path.write_bytes(pickle.dumps(embedding_dict))

    def read_embeddings(self):
        if not self.embedding_path.exists():
            logger.warning(
                f'Attempting to access non-existent embeddings at {self.embedding_path}'
            )
            return None

        return pickle.loads(self.embedding_path.read_bytes())

    @property
    def model(self):
        if self._model is None:
            # model trained with 128 token seqs
            self._model = SentenceTransformer('sentence-transformers/all-MiniLM-L12-v2')
        return self._model

    @property
    def embeddings(self):
        if self._embeddings is None:
            self._embeddings = self.read_embeddings()
        return self._embeddings

    def embed_chunks(self, chunks, batch_size=64, show_prog=True):
        return self.model.encode(
            chunks,
            batch_size           = batch_size,
            show_progress_bar    = show_prog,
            convert_to_numpy     = True,
            normalize_embeddings = True
        )

        self,
        query      : str,
        index_name : str,
        limit      : int = 10,
        score_threshold = 0.5,
    ):
        '''
            index_name: one of ['chunks','blocks','notes']
        '''
        if not query:

        if index_name not in self.embeddings:
                f'Index "{index_name}" does not exist'
            )

        start = time.time()

        query_embedding = self.embed_chunks(query, show_prog=False)
        index_ids, index_embeddings, index_items = self.embeddings[index_name]

        hits = util.semantic_search(
            query_embedding,
            index_embeddings,
            top_k=limit,
            score_function=util.dot_score
        )[0]

        hits = [hit for hit in hits if hit['score'] >= score_threshold]

        for hit in hits:
            idx               = hit['corpus_id']
            hit['group_name'] = index_ids[idx]
            hit['item']       = index_items[idx]

        logger.info(f'{len(hits)} hits in {time.time()-start:.2f}s')

        return hits

'''
CO3

'''

from functools import wraps, partial

def collate(action_key, action_groups=None):
    def decorator(func):
        nonlocal action_groups
        
        if action_groups is None:
            action_groups = [None]
        func._action_data = (action_key, action_groups)
        return func
    return decorator

class FormatRegistryMeta(type):
    def __new__(cls, name, bases, attrs):
        action_registry = {}
        group_registry  = defaultdict(list)

        def register_action(method):
            nonlocal action_registry, group_registry

            if hasattr(method, '_action_data'):
                action_key, action_groups = method._action_data
                action_registry[action_key] = (method, action_groups)

                for action_group in action_groups:
                    group_registry[action_group].append(action_key)

        # add registered superclass methods; iterate over bases (usually just one), then
        # that base's chain down (reversed), then methods from each subclass
        for base in bases:
            for _class in reversed(base.mro()):
                methods = inspect.getmembers(_class, predicate=inspect.isfunction)
                for _, method in methods:
                    register_action(method)

        # add final registered formats for the current class, overwriting any found in
        # superclass chain
        for attr_name, attr_value in attrs.items():
            register_action(attr_value)

        attrs['action_registry'] = action_registry
        attrs['group_registry']  = group_registry

        return super().__new__(cls, name, bases, attrs)

class CO3(metaclass=FormatRegistryMeta):
    '''
    Conversion & DB insertion base class

    CO3: COllate, COllect, COmpose

    - Collate: organize and transform conversion outputs, possibly across class components
    - Collect: gather core attributes, conversion data, and subcomponents for DB insertion
    - Compose: construct object-associated DB table references through the class hierarchy

    .. admonition:: on action groups

        Group keys are simply named collections to make it easy for storage components to
        be attached to action subsets. They do _not_ augment the action registration
        namespace, meaning the action key should still be unique; the group key is purely
        auxiliary.

        Action methods can also be attached to several groups, in case there is
        overlapping utility within or across schemas or storage media. In this case, it
        becomes particularly critical to ensure registered ``collate`` methods really are
        just "gathering results" from possibly heavy-duty operations, rather than
        performing them when called, so as to reduce wasted computation.
    '''
    @property
    def attributes(self):
        '''
        Method to define how a subtype's inserts should be handled under ``collect`` for
        canonical attributes, i.e., inserts to the type's table.
        '''
        return vars(self)

    @property
    def components(self):
        '''
        constituent components that need handling.
        '''
        return []

    def collation_attributes(self, action_key, action_group):
        '''
        Return "connective" collation component data, possibly dependent on
        instance-specific attributes and the action arguments. This is typically the
        auxiliary structure that may be needed to attach to responses from registered
        ``collate`` calls to complete inserts.

        Note: this method is primarily used by ``Mapper.collect()``, and is called just
        prior to collector send-off for collation inserts and injected alongside collation
        data. Common structure in collation components can make this function easy to
        define, independent of action group for instance.
        '''
        return {}

    def collate(self, action_key, *action_args, **action_kwargs):
        if action_key not in self.action_registry:
            logger.debug(f'Collation for {action_key} not supported')
        else:
            action_method = self.action_registry[action_key][0]
            return action_method(self, *action_args, **action_kwargs)

'''
Defines the Collector base class.

Note:
    Despite the structure of the database module, this class does not currently inherit
    from a super class in localsys.db (like the accessors and managers, for instance).
    This will likely ultimately be the model that's embraced, but until FTS (or other
    groups) need a collector, this will be remain an independent class. It is, however,
    named like a concrete subclass, taking on the "Core" prefix.
'''

from uuid import uuid4

from co3.schema import Schema

class Collector[C: Component]:
    def __init__(self, schema: Schema[C]):
        self.schema = schema

        self._inserts = {}

    @property
    def inserts(self):
        return self._inserts_from_receipts()

    def _inserts_from_receipts(self, receipts: list[str]|None=None, pop=False):
        '''
        Group up added inserts by Component, often to be used directly for bulk insertion.
        Optionally provide a list of ``receipts`` to group up only the corresponding subset of
        inserts, and ``pop`` to remove encountered receipts from the internal store.
        '''
        inserts = defaultdict(list)
        
        if receipts is None:
            receipts = list(self._inserts.keys())

        for receipt in receipts:
            if pop:
                receipt_tuple = self._inserts.pop(receipt, None)
            else:
                receipt_tuple = self._inserts.get(receipt, None)

            if receipt_tuple is not None:
                component, insert_data = receipt_tuple
                inserts[component].append(insert_data)

        return dict(inserts)

    def _reset_session(self):
        self._inserts = defaultdict(lambda: defaultdict(list))

    def _generate_unique_receipt(self):
        receipt = str(uuid4())
        while receipt in self._inserts:
            receipt = str(uuid4())

        return receipt

    def add_insert(
        self,
        component   : C,
        insert_data : dict,
        receipts    : list | None = None,
    ):
        '''
            component:   Component from registered schema
            insert_data: dict with (possibly raw/incomplete) insert data
            receipts:    optional list to which generated receipt should be appended.
                         Accommodates the common receipt list aggregation pattern.
        '''
        if component not in self.schema:
            #logger.debug(f'Inserts provided for non-existent table {table_name}')

        receipt = self._generate_unique_receipt()
        self._inserts[receipt] = (
            component,
            component.prepare_insert_data(insert_data),
        )

        if receipts is not None:
            receipts.append(receipt)

    def collect_inserts(self, receipts:list[str]|None=None):
        '''
        Collect insert-ready dictionaries for the core primitive schema. This method is
        effectively a light wrapper around the File and Note-based collection logic
        elsewhere in the class. 

        The overall collection scheme embraces a session-like sequential update model to
        an internal insert tracker. The sequence of insert methods is ordered according to
        the schema hierarchy, and higher level inserts dictate the scope for lower level
        inserts (all methods check and populate the same ``inserts`` dictionary). Calling
        this method flushes any existing inserts, ensuring a re-scan takes place across
        calls (or "sessions").

            skip_updated: whether to ignore primitives with existing up-to-date
                          database entries

        Returns:
            Table name-indexed dictionary of insert lists (of column name-indexed dicts)
        '''
        return self._inserts_from_receipts(receipts, pop=True)

'''
Component

'''

class Component[T]:
    def __init__(self, name, obj: T):
        self.name = name
        self.obj  = obj

    def __str__(self):
        return f'<Component ({self.__class__.__name__})> {self.name}'

    def __repr__(self):

    def get_attributes(self):

'''
Dev note:
    Any reason to have ComposeableComponents and Relations as separate types? The thought
    is that there may be some possible Component types we want to be able to Compose that
    wouldn't logically be Relations. But the gap here might be quite small
'''

from typing import Self

from co3.util.types import SQLTableLike

class ComposableComponent[T](Component[T], metaclass=ABCMeta):
    '''
    Components that can be composed with others of the same type.
    '''
    def compose(self, component: Self, on, outer=False) -> Self:
        '''
        Abstract composition.
        '''

# relational databases
class Relation[T](ComposableComponent[T]):
    '''
    Relation base for tabular components to be used in relation DB settings. Attempts to
    adhere to the set-theoretic base outlined in the relational model [1]. Some
    terminology:

    Relation: table-like container
    | -> Heading: set of attributes
    |    | -> Attribute: column name
    | -> Body: set of tuples with domain matching the heading
    |    | -> Tuple: collection of values

    [1]: https://en.wikipedia.org/wiki/Relational_model#Set-theoretic_formulation

    Note: development tasks
        As it stands, the Relation skeleton is incredibly lax compared to the properties and
        operations that should be formally available, according its pure relational algebra
        analog. 

        Relations are also generic up to a type T, which ultimately serves as the base object
        for Relation instances. We aren't attempting to implement some generally useful
        table-like class here; instead we're just exposing a lightweight interface that's
        needed for a few CO3 contexts, and commonly off-loading most of the heavy-lifting to
        true relation objects like SQLAlchemy tables.
    '''
    def compose(
        self,
        _with: Self,
        on,
        outer=False
    ):
        return self

class SQLTable(Relation[SQLTableLike]):
    @classmethod
    def from_table(cls, table: sa.Table):
        '''
        Note that the sa.Table type is intentional here; not all matching types for
        SQLTableLike have a defined ``name`` property
        '''
        return cls(table.name, table)

    def get_attributes(self) -> tuple:
        return tuple(self.obj.columns)

    def get_column_defaults(self, include_all=True):
        '''
        Provide column:default pairs for a provided SQLAlchemy table.

            include_all: whether to include all columns, even those without explicit defaults
        '''
        default_values = {}
        for column in self.get_attributes():
            if column.default is not None:
                default_values[column.name] = column.default.arg
            elif column.nullable:
                default_values[column.name] = None
            else:
                # assume empty string if include_all and col has no explicit default 
                # and isn't nullable
                if include_all and column.name != 'id':
                    default_values[column.name] = ''

        return default_values

    def prepare_insert_data(self, insert_data: dict) -> dict:
        '''
        Modifies insert dictionary with full table column defaults
        '''
        insert_dict = self.get_column_defaults()
        insert_dict.update(
            { k:v for k,v in insert_data.items() if k in insert_dict }
        )

        return insert_dict

    def compose(self, _with: Self, on, outer=False):
        return self.__class__(
            f'{self.name}+{_with.name}',
            self.obj.join(_with.obj, on, isouter=outer)
        )

# key-value stores
class Dictionary(Relation[dict]):
        return tuple(self.obj.keys())

# document databases
class Document[T](Component[T]):
    pass

# graph databases
class Node[T](Component[T]):
    pass

'''
Database

objects.

    with db.engine.connect() as connection:
        db.access.select(
            <query>
        )
        db.manager.insert(
            data
        )

    db.select(<query>)
    
    db.insert(<query>, data)

.. admonition:: on explicit connection contexts

    Older models supported Accessors/Managers that housed their own Engine instances, and
    when performing actions like ``insert``, the Engine would be passed all the way through
    until a Connection could be spawned, and in that context the single action would be
    made. This model forfeits a lot of connection control, preventing multiple actions
    under a single connection.

    The newer model now avoids directly allowing Managers/Accessors access to their own
    engines, and instead they expose methods that explicitly require Connection objects.
    This means a user can invoke these methods in their own Connection contexts (seen
    above) and group up operations as they please, reducing overhead. The Database then
    wraps up a few single-operation contexts where outer connection control is not needed.
'''

from co3.manager  import Manager
from co3.indexer  import Indexer
from co3.engine   import Engine
from co3.schema   import Schema

class Database[C: Component]:
    '''
    Generic Database definition

    Generic to both a Component (C), and an Engine resource type (R). The Engine's
    generic openness must be propagated here, as it's intended to be fully abstracted away
    under the Database roof. Note that we cannot explicitly use an Engine type in its
    place, as it obscures its internal resource type dependence when we need it for
    hinting here in ``__init__``.

    .. admonition:: Development TODO list

        Decide on official ruling for assigning Schema objects, and verifying any
        attempted Component-based actions (e.g., inserts, selects) to belong to or be a
        composition of Components within an attached Schema. Reasons for: helps complete
        the sense of a "Database" here programmatically, incorporating a more structurally
        accurate representation of allowed operations, and prevent possible attribute and
        type collisions. Reasons against: generally not a huge concern to align Schemas as
        transactions will rollback, broadly increases a bit of bulk, and users often
        expected know which components belong to a particular DB. Leaning more to **for**,
        and would only apply to the directly supported method passthroughs (and thus would
        have no impact on independent methods like ``Accessor.raw_select``). Additionally,
        even if component clashes don't pose serious risk, it can be helpful to
        systematically address the cases where a misalignment is occurring (by having
        helpful ``verify`` methods that can be ran before any actions).
    '''
    _accessor_cls: type[Accessor[C]] = Accessor[C]
    _manager_cls:  type[Manager[C]]  = Manager[C]
    _engine_cls:   type[Engine]      = Engine

    def __init__(self, *engine_args, **engine_kwargs):
        '''
            engine_args: positional arguments to pass on to the Engine object during
                         instantiation 
            engine_kwargs: keyword arguments to pass on to the Engine object during
                           instantiation 

        Variables:
            _local_cache: a database-local property store for ad-hoc CacheBlock-esque
                          methods, that are nevertheless _not_ query/group-by responses to
                          pass on to the Indexer. Dependent properties should write to the
                          this cache and check for existence of stored results; the cache
                          state must be managed globally.
        ''' 
        self.engine = self._engine_cls(*engine_args, **engine_kwargs)

        self.accessor = self._accessor_cls()
        self.manager  = self._manager_cls()
        self.indexer  = Indexer(self.accessor)

        self._local_cache = {}
        self._reset_cache = False

    def select(self, component: C, *args, **kwargs):
        '''
        .. admonition:: Dev note

            args and kwargs have to be general/unspecified here due to the possible
            passthrough method adopting arbitrary parameters in subtypes. I could simply
            overload this method in the relevant inheriting DBs (i.e., by matching the
            expected Accessor's .select signature).
        '''
        with self.engine.connect() as connection:
            return self.accessor.select(
                connection,
                component,
                *args,
                **kwargs
            )

    def insert(self, component: C, *args, **kwargs):
            return self.manager.insert(
                *args,
                **kwargs
            )

    def recreate(self, schema: Schema[C]):
        self.manager.recreate(schema, self.engine)

    @property
    def index(self):
        if self.reset_cache:
            self._index.cache_clear()
            self.reset_cache = False
        return self._index

    @property
    def manage(self):
        '''
        Accessing ``.manage`` queues a cache clear on the external index, as well wipes the
        local index.
        '''
        self.reset_cache = True
        return self._manage

    def populate_indexes(self): pass

from co3.database import Database

from co3.accessors.fts import FTSAccessor
from co3.managers.fts  import FTSManager

class FTSDatabase(Database):
    accessor = FTSAccessor
    manager  = FTSManager

from co3.database import Database, Engine

from co3.accessors.sql import RelationalAccessor, SQLAccessor
from co3.managers.sql  import RelationalManager,  SQLManager

class RelationalDatabase[C: RelationR](Database):
    '''
    accessor/manager assignments satisfy supertype's type settings;
    ``TabluarAccessor[Self, C]`` is of type ``type[RelationalAccessor[Self, C]]``
    (and yes, ``type[]`` specifies that the variable is itself being set to a type or a
    class, rather than a satisfying _instance_)
    '''
    _accessor_cls: type[RelationalAccessor[C]] = RelationalAccessor[C]
    _manager_cls:  type[RelationalManager[C]]  = RelationalManager[C]

class SQLDatabase[C: SQLTable](RelationalDatabase[C]):
    _accessor_cls = SQLAccessor
    _manager_cls  = SQLManager
    _engine_cls   = SQLEngine

class SQLiteDatabase(SQLDatabase[SQLTable]):
    pass

from co3.accessors.vss import VSSAccessor
from co3.managers.vss  import VSSManager

class VSSDatabase(Database):
    accessor = VSSAccessor
    manager  = VSSManager

from contextlib import contextmanager

class Engine:
    '''
    Engine base class. Used primarily as a Database connection manager, with general
    methods that can be defined for several kinds of value stores.

    Note that this is where the connection hierarchy is supposed to stop. While some
    derivative Engines, like SQLEngine, mostly just wrap another engine-like object, this
    is not the rule. That is, inheriting Engine subtypes shouldn't necessarily expect to
    rely on another object per se, and if such an object is required, _this_ is the class
    is meant to be skeleton to supports its creation (and not merely a wrapper for some
    other type, although it may appear that way when such a type is in fact readily
    available).

    .. admonition:: why is this object necessary?

        More specifically, why not just have all the functionality here packed into the
        Database by default? The answer is that, realistically, it could be. The type
        separation between the Engine and Database is perhaps the least substantiated in
        CO3. That being said, it still serves a purpose: to make composition of subtypes
        easier. The Engine is a very lightweight abstraction, but some Engine subtypes
        (e.g., FileEngines) may be used across several sibling Database types. In this
        case, we'd have to repeat the Engine-related functionality for such sibling types.
        Depending instead on a singular, outside object keeps things DRY. If Databases and
        Engines were uniquely attached type-wise 1-to-1 (i.e., unique Engine type per
        unique Database type), a separate object here would indeed be a waste, as is the
        case for any compositional typing scheme.

    .. admonition:: dev note

        This class is now non-generic. It was originally conceived as a generic, depending
        on a "resource spec type" to be help define expected types on initialization.
        This simply proved too messy, required generic type propagation to the Database
        definition, and muddied the otherwise simple args and kwargs forwarding for
        internal manager creation.
    '''
    def __init__(self, *manager_args, **manager_kwargs):
        self._manager        = None
        self._manager_args   = manager_args
        self._manager_kwargs = manager_kwargs

    @property
    def manager(self):
        '''
        Return Engine's singleton manager, initializing when the first call is made.
        '''
        if self._manager is None:
            self._manager = self._create_manager()

        return self._manager

    def _create_manager(self):
        '''
        Create the session manager needed for connection contexts. This method is called
        once by the ``.manager`` property function when it is first accessed. This method is
        separated to isolate the creation logic in inheriting types.

        Note that this method takes no explicit arguments. This is primarily because the
        standard means of invocation (the manager property) is meant to remain generally
        useful here in the base class, and can't be aware of any specific properties that
        might be extracted in subtype initialization. As a result, we don't even try to
        pass args, although it would just look like a forwarding of the readily manager
        args and kwargs anyhow. As such, this method should make direct use of these
        instance variables as needed.
        '''
        
    @contextmanager
    def connect(self, timeout=None):
        '''
        Open a connection to the database specified by the resource. Exactly what the
        returned connection looks like remains relatively unconstrained given the wide
        variety of possible database interactions. This function should be invoked in
        with-statement contexts, constituting an "interaction session" with the database
        (i.e., allowing several actions to be performed using the same connection).
        '''

from co3.engine import Engine

class SQLEngine(Engine):
    def __init__(self, url: str | sa.URL, **kwargs):
        super().__init__(url, **kwargs)

        return sa.create_engine(*self._manager_args, **self._manager_kwargs)
        
        return self.manager.connect()
    
    def execute(
        statement,
    ):
        '''
        Execute a general SQLAlchemy statement, optionally binding provided parameters and
        returning associated column names.

            connection:   database connection instance
            statement:    SQLAlchemy statement
            bind_params: 
            include_cols: whether to return
        '''
        res = connection.execute(statement, bind_params)

        if include_cols:
            cols = list(res.mappings().keys())
            return res, cols

        return res

    def exec_explicit(connection, statement, bind_params=None):
        trans = connection.begin()  # start a new transaction explicitly
        try:
            result = connection.execute(statement, bind_params)
            trans.commit()  # commit the transaction explicitly
            return result
        except:
            trans.rollback()  # rollback the transaction explicitly
            raise

import threading

class Indexer:
    '''
    Indexer class

    Provides restricted access to an underlying Accessor to enable more efficient, superficial
    caching.

    Cache clearing is to be handled by a wrapper class, like the Database.

    Caching occurs at the class level, with indexes prefixed by table's origin Composer.
    This means that cached selects/group-bys will be available regardless of the provided
    Accessors so long as the same Composer is used under the hood. 
    '''
    _cls_select_cache  = {}
    _cls_groupby_cache = defaultdict(dict)
    
    def __init__(self, accessor, cache_select=True, cache_groupby=True):
        self.accessor = accessor

        # set instance caches; if remains None, methods can't index
        self._select_cache = None
        self._groupby_cache = None

        if cache_groupby and not cache_select:
            raise ValueError('cannot cache groupbys without select caching enabled')

        if cache_select:
            self._select_cache = self._cls_select_cache

        if cache_groupby:
            self._groupby_cache = self._cls_groupby_cache

        self._access_lock = threading.Lock()

    def cache_clear(self, group_by_only=False): 
        self._groupby_cache.clear()
        if not group_by_only:
            self._select_cache.clear()

    def cache_block(
        self,
        table,
        **kwargs,
    ):
        '''
        Provide a user-friendly, dynamically re-indexable
        '''
        return CacheBlock(
            indexer = self,
            table   = table,
            **kwargs,
        )

    def cached_query(
        self,
        table, 

        cols        = None,
        where       = None,
        distinct_on = None,
        order_by    = None,
        limit       = 0,

        group_by    = None,
        agg_on      = None,
        index_on    = None,
    ):
        '''
        Like ``group_by``, but makes a full query to the Accessors table ``table_name`` and
        caches the results. The processing performed by the GROUP BY is also cached.

        Update: ``cached_select`` and ``cached_group_by`` now unified by a single
        ``cached_query`` method. This allows better defined GROUP BY caches, that are
        reactive to the full set of parameters returning the result set (and not just the
        table, requiring a full query).

        Note: on cache keys
            Cache keys are now fully stringified, as many objects are now allowed to be
            native SQLAlchemy objects. Indexing these objects works, but doing so will
            condition the cache on their memory addresses, which isn't what we want.
            SQLAlchemy converts most join/column/table-like objects to reasonable strings,
            which will look the same regardless of instance.

            Context: this became a clear issue when passing in more
            ``order_by=<col>.desc()``. The ``desc()`` causes the index to store the column in
            an instance-specific way, rather than an easily re-usable, canonical column
            reference. Each time the CoreDatabase.files() was being called, for instance,
            that @property would be re-evaluated, causing ``desc()`` to be re-initialized,
            and thus look different to the cache. Stringifying everything prevents this
            (although this could well be an indication that only a single ``cache_block``
            should ever be returned be database properties).

        Note: on access locks
            A double-checked locking scheme is employed before both of the stages (select
            and manual group by), using the same lock. This resolves the common scenario
            where many threads need to look up a query in the cache, experience a cache
            miss, and try to do the work. This non-linearly explodes the total time to
            wait in my experience, so doing this only when needed saves tons of time,
            especially in high-congestion moments.
        '''
        start     = time.time()
        cache_key = tuple(map(str, (table, cols, where, distinct_on, order_by, limit)))

        # apparently this is the double-check locking scheme (didn't realize when implementing)
        if self._select_cache is None or cache_key not in self._select_cache:
            # cache re-compute possible, acquire lock to continue. A later thread may
            # acquire this after work has been done by an earlier thread, so re-eval the
            # condition below before actually performing a DB read. If access results in a
            # cache hit, locking isn't needed.
            with self._access_lock:
                if self._select_cache is None or cache_key not in self._select_cache:
                    results = self.accessor.select(
                        table,
                        cols=cols,
                        where=where,
                        distinct_on=distinct_on,
                        order_by=order_by,
                        limit=limit,
                        mappings=True
                    )

                    # cache results if select_cache is defined
                    if self._select_cache is not None: 
                        self._select_cache[cache_key] = results

                    logger.debug(
                        f'Indexer "select" cache miss for table "{table}": access in {time.time()-start:.4f}s'
                    )
                else:
                    results = self._select_cache[cache_key]
                        f'Indexer "select" cache hit for table "{table}": access in {time.time()-start:.4f}s'
                    )
        else:
            results = self._select_cache[cache_key]
            logger.debug(
                f'Indexer "select" cache hit for table "{table}": access in {time.time()-start:.4f}s'
            )

        cache_key = (*cache_key, group_by, agg_on, index_on)

        if group_by is not None:
            if self._groupby_cache is None or cache_key not in self._groupby_cache:
                with self._access_lock:
                    if self._groupby_cache is None or cache_key not in self._groupby_cache:
                        results = self.group_by(
                            results,
                            group_by     = group_by,
                            agg_on       = agg_on,
                            index_on     = index_on,
                            return_index = True,
                        )

                        if self._groupby_cache is not None:
                            self._groupby_cache[cache_key] = results

                        logger.debug(
                            f'Indexer "group_by" cache miss for table "{table}": access in {time.time()-start:.4f}s'
                        )
                    else:
                        results = self._groupby_cache[cache_key]
                            f'Indexer "group_by" cache hit for table "{table}": access in {time.time()-start:.4f}s'
                        )
            else:
                results = self._groupby_cache[cache_key]
                logger.debug(
                    f'Indexer "group_by" cache hit for table "{table}": access in {time.time()-start:.4f}s'
                )

        return results
        
    def group_by(
        cls,
        rows,
        group_by,
        agg_on=None,
        index_on=None,
        return_index=False,
    ):
        '''
        Post-query "group by"-like aggregation. Creates an index over a set of columns
        (``group_by_cols``), and aggregates values from ``agg_cols`` under the groups.

        Rows can be dicts or mappings, and columns can be strings or SQLAlchemy columns.
        To ensure the right columns are being used for the operation, it's best to pass in
        mappings and use SQA columns if you aren't sure exactly how the keys look in your
        results (dicts can have ambiguous keys across tables with the same columns and/or
        different labeling schemes altogether).

        TODO: add a flag that handles None's as distinct. That is, for the group_by
        column(s) of interest, if rows in the provided query set have NULL values for
        these columns, treat all such rows as their "own group" and return them alongside
        the grouped/aggregated ones. This is behavior desired by something like
        FTSManager.recreate(), which wants to bundle up conversions for blocks
        (effectively grouping by blocks.name and link.id, aggregating on
        block_conversions.format, then flattening). You could either do this, or as the
        caller just make sure to first filter the result set before grouping (e.g.,
        splitting the NULL-valued rows from those that are well-defined), and then
        stitching the two sets back together afterward.

        Multi-dim update:
        
        - group_by: can be a tuple of tuples of columns. Each inner tuple is a nested
          "group by index" in the group by index
        - 
        '''
        if not rows:
            return {} if return_index else []

        rows_are_mappings = not isinstance(rows[0], dict)

        if not rows_are_mappings:
            if isinstance(group_by, sa.Column):
                group_by = group_by.name
            else:
                group_by = str(group_by)

        #if group_by is None:                     group_by = []
        #elif not isinstance(group_by, Iterable): group_by = [group_by]

        if agg_on is None:                     agg_on = []
        elif not isinstance(agg_on, Iterable): agg_on = [agg_on]

        if index_on is None:                     index_on = []
        elif not isinstance(index_on, Iterable): index_on = [index_on]

        agg_on_names = []
        for agg in agg_on:
            # if a SQA column, can either use `.name` or `str(c)`. The latter includes the
            # table name, the former doesn't; ambiguity can be introduced here.
            if isinstance(agg, sa.Column):
                agg_on_names.append(agg.name)
            else:
                agg_on_names.append(str(agg))

        index_on_names = []
        for index in index_on:
            if isinstance(index, sa.Column):
                index_on_names.append(index.name)
            else:
                index_on_names.append(str(index))

        # when rows are dicts, use columns' string names
            agg_on = agg_on_names
            index_on = index_on_names

        #print(f'rows_are_mappings: {rows_are_mappings}')
        #print(f'group_by: {group_by}')
        #print(f'agg_on: {agg_on}')
        #print(f'agg_on_names: {agg_on_names}')
        #print(f'index_on: {index_on}')
        #print(f'index_on_names: {index_on_names}')

        for row in rows:
            # generic get

            # wrap possible mapping dict
            row_dict = dict(row)

            # add new entries; standardize 
            #aggregates = {}
            #for agg_name in agg_on_names:
            #    aggregates[agg_name] = []
            #row_dict['aggregates'] = aggregates
            row_dict['aggregates'] = []

            indexes = {}
            for index_name in index_on_names:
                indexes[index_name] = {}
            row_dict['indexes'] = indexes

                continue

                group_by_idx[group_by_attr] = row_dict

            # actually include all agg cols, even if None, so agg array indexes align
            agg_dict = {
                agg_on_names[i] : row.get(agg_col)
                for i, agg_col in enumerate(agg_on)
            }

            #aggregates = group_by_idx[group_by_attr]['aggregates']
            #for agg_key, agg_val in agg_dict.items():
            #    aggregates[agg_key].append(agg_val)
            aggregates = group_by_idx[group_by_attr]['aggregates']
            aggregates.append(agg_dict)

            indexes = group_by_idx[group_by_attr]['indexes']
            for i, index_col in enumerate(index_on):
                index_name = index_on_names[i]
                indexes[index_name][row[index_col]] = agg_dict

        if return_index:
            return group_by_idx

        return list(group_by_idx.values())

class CacheBlock:
    '''
    Wraps up a set of query parameters for a specific entity, and provides cached access
    to different types of "re-queries" via an associated Indexer.

    .. admonition:: Additional details

        The goal here is to help build/define entities as the possibly complex
        transformations on the base schema that they are. For example, the Note primitive
        (entity) incorporates details across ``files``, ``notes``, ``note_conversions``,
        and ``note_conversion_matter`` tables (defined in a single endpoint by a
        Composer), often needs to be selected in particular ways (via an Accessor), and
        results stored for fast access later on (handled by an Indexer). This pipeline can
        be daunting and requires too many moving parts to be handled explicitly
        everywhere. CacheBlocks wrap up a set of query "preferences," exposing a simpler
        interface for downstream access to entities. It still allows for low-level control
        over re-grouping/indexing, raw hits to the actual DB, etc, but keeps things
        tighter and well-behaved for the Indexer.
        
        You can think of these as the Indexer's "fingers"; they're deployable mini-Indexes
        that "send back" results to the class cache, which is "broadcast" to all other
        instances for use when necessary.

    .. admonition:: Example usage
        
        .. code-block:: python

            cb = CacheBlock()

            # Set up cached queries with chained params or via call:
            
            cb.where(t.notes.c.name=="name").group_by(t.note_conversions.c.format)
            cb() # get results

            # - OR - # (use strings when known)

            cb.where(t.notes.c.name=="name").group_by('format')

            # - OR - # (use kwargs in the call; results returned right away)

            cb(
                where=(t.notes.c.name=="name"),
                group_by='format'
            )
    '''
    def __init__(
        self,
        indexer,
        table,

        limit       = 0,

    ):
        self.indexer = indexer

        self.query_args = {
            'table'       : table,
    
            'cols'        : cols,
            'where'       : where,
            'distinct_on' : distinct_on,
            'order_by'    : order_by,
            'limit'       : limit,
    
            'group_by'    : group_by,
            'agg_on'      : agg_on,
            'index_on'    : index_on,
        }

    def _query(self, **kwargs):
        '''Make cached query with defaults, override with those provided'''
        return self.indexer.cached_query(**{
            k : (v if k not in kwargs else kwargs[k])
            for k,v in self.query_args.items()
        })

    def __call__(self, **kwargs):
        '''
        TODO: overload this for the queries, i.e. getting keys or returning aggregates
        '''
        return self._query(**kwargs)

    def where(self, where):
        self.query_args['where'] = where
        #return self._query(where=where)

    def distinct_on(self, distinct_on):
        self.query_args['distinct_on'] = distinct_on
        #return self._query(distinct_on=distinct_on)

    def order_by(self, order_by):
        self.query_args['order_by'] = order_by
        #return self._query(order_by=order_by)

    def limit(self, limit):
        self.query_args['limit'] = limit

    def group_by(self, group_by):
        self.query_args['group_by'] = group_by
        #return self._query(group_by=group_by)
    
'''
Manager

'''

class Manager[C: Component](metaclass=ABCMeta):
    '''
    Management wrapper class for table groupings.

    The schema is centered around a connected group of tables (via foreign keys). Thus,
    most operations need to be coordinated across tables. A few common operations are
    wrapped up in this class to then also be mirrored for the FTS counterparts.
    '''
    def recreate(self, schema: Schema[C], engine: Engine):

    def sync(self):

    def migrate(self):

from tqdm import tqdm

from co3.manager import Manager
from co3.accessors.sql import SQLAccessor

class FTSManager(Manager):
    def __init__(self, database, engine=None):
        super().__init__(database, engine)

        self.accessor = SQLAccessor(engine)
        self.composer = CoreComposer()

    def recreate_simple(self):
        select_cols = [
            tables.files.c.rpath, tables.files.c.ftype,
            
            tables.notes.c.title, tables.notes.c.link,

            tables.note_conversion_matter.c.type,
            
            tables.blocks.c.name, tables.blocks.c.header,
            tables.blocks.c.elem_pos,
            
            tables.block_conversion_matter.c.content,

            tables.links.c.target,
            #tables.links.c.target.label('link_target'),
        ]

        inserts, res_cols = self.accessor.select(
            'leftward_link_conversions',
            cols=select_cols,
            include_cols=True,
        )

        util.db.populate_fts5(
            self.engine,
            ['search'],
            #columns=select_cols,
            columns=res_cols,
            inserts=inserts,
        )

    def recreate_pivot(self):
        block_cols = [
            
            #tables.block_conversion_matter.c.content,

            #tables.links.c.target,

        ]

        lcp = self.accessor.link_content_pivot(cols=block_cols)

        inserts, cols = self.accessor.select(
            lcp,
            #cols=select_cols
        )
        logger.info(f'FTS recreate: pre-index SELECT took {time.time()-start:.2f}s')

        return inserts

        #util.db.populate_fts5(
        #    self.engine,
        #    ['search'],
        #    #columns=select_cols,
        #    columns=res_cols,
        #    inserts=inserts,
        #)

    def recreate(self):
            tables.links.c.id,

            tables.files.c.title, tables.files.c.link,

            tables.notes.c.type,

            tables.elements.c.name, tables.blocks.c.header,
            tables.elements.c.elem_pos,

            tables.element_conversions.c.format,
            tables.element_conversions.c.content,

            tables.links.c.type.label('link_type'),
        ]

        fts_basis = self.composer.get_table('leftward_link_conversions')
        inserts = self.accessor.select(
            fts_basis,
        )

        flattened_blocks = defaultdict(lambda:defaultdict(lambda:None))
        other_results = []

        # flatten conversions across supported formats
        # i.e. put "src" and "html" into columns within a single row
        for row in inserts:
            name = row['name']
            
            # remove pure identifier rows
            link_id = row.pop('id')
            _format = row.pop('format')
            content = row.pop('content')
            
            # add new derived rows
            row['src']  = None
            row['html'] = None
            
            if name is None:
                other_results.append(row)
                continue
            
            if flattened_blocks[name][link_id] is None:
                flattened_blocks[name][link_id] = row
                
            nrow = flattened_blocks[name][link_id]
            if _format == 'src':
                 nrow['src'] = content
            elif _format == 'html5':
                 nrow['html'] = content
                    
        inserts = other_results
        for name, link_dict in flattened_blocks.items():
            for link_id, row in link_dict.items():
                inserts.append(row)

        # should have at least one result to index
        assert len(inserts) > 0
        cols = inserts[0].keys()

        # all keys should match
        assert all([cols == d.keys() for d in inserts])

        logger.info(f'FTS recreate: insert post-processing took {time.time()-start:.2f}s')

            columns=list(cols),
        )

    def update(self): pass

    def sync(self): pass

    def migrate(self): pass

def recreate_old(self):
    full_table = schema.leftward_block_conversions()
    primitive_tables = schema.primitive_tables() 

    virtual_inserts = []
    for ptype, table in primitive_tables.items():
        table_rows = db.util.sa_exec_dicts(self.engine, sa.select(table))
        for row in table_rows:
            pdict = util.db.prepare_insert(full_table, row)
            pdict['ptype'] = ptype

            virtual_inserts.append(pdict)

    select_cols = [
        tables.blocks.c.name, tables.blocks.c.content,
        tables.blocks.c.html, tables.blocks.c.header,
        tables.blocks.c.block_pos, tables.blocks.c.note_name,
        
        tables.links.c.to_file,
        
        tables.notes.c.link,
        
        tables.note_conversion_matter.c.type,
        tables.note_conversion_matter.c.summary,

        'ptype',
    ]
        
    util.db.populate_fts5(
        self.engine,
        'search',
        columns=select_cols,
        inserts=virtual_inserts,
    )
        
'''
    - Tables with unique constraints have been equipped with ``sqlite_on_conflict_unique``
      flags, enabling conflicting bulk inserts to replace conflicting rows gracefully. No
      need to worry about explicitly handling upserts.
    - The bulk insert via conn.execute(<insert>,<row_list>) automatically ignores
      irrelevant column names within provided record dicts, whereas explicit .values() calls
      for non-bulk inserts will throw errors if not aligned perfectly. We want to keep these
      bulk calls AND allow update/replacements on conflict; the setting above allows this
      bulk usage to continue as is.

    1. Collect all possible update objects, and use a SQLite bulk insert that only updates
       if needed (based on modified time, for example). This sounds on the surface like
       the right approach since we defer as much to bulk SQL logic, but it's far and away
       the worse option b/c prepping all file/note/etc objects is too expensive to
       ultimately throw away when finding out an update isn't needed. For example, if we
       wanted to perform note updates _inside_ the SQL call (like an ``INSERT .. UPDATE ..
       IF``, as opposed to determining the objects to insert _outside_ of the SQL call),
       you would need to bring each of the note's HTML prior to the insert check. There's
       no 2-stage processing here where you can check if the note needs to be converted
       b/c it's out of date, and only then perform the computation.
    2. Instantiate objects sequentially, each time checking with the DB to see if full
       processing is needed. This makes much more sense when the full processing is very
       expensive, as it is with Note conversion. This would iterate through available notes,
       perform a ``SELECT`` on the target table to see if the note needs updating, and if so
       perform the remaining computation. Those objects then get added to a "update object
       list" to be inserted in bulk, but you make sequential ``SELECT`` checks before that.
    
       The one extra optimization you could maybe make here is doing a full SELECT on the
       target table and bring all rows into memory before iterating through the objects.
       This would likely make it faster than whatever SQLAlchemy overhead there may be. It
       also might just be outright required given Connection objects aren't thread-safe;
       we can get away with single thread global SELECT, threaded checking during object
       build, then single thread bulk INSERT. (**Note**: this is what the method does).
'''

from concurrent.futures import wait, as_completed

from tqdm.auto import tqdm

#from localsys.reloader.router._base import ChainRouter, Event

class RelationalManager[R: Relation](Manager[R]):
    pass

class SQLManager(RelationalManager[SQLTable]):
    '''
    Core schema table manager. Exposes common operations and facilitates joint operations
    needed for highly connected schemas.

    In particular, Managers expose insertion abstractions that take table-indexed groups
    of rows and bundle them under a single transaction. This is important for table groups
    with foreign keys and cascading deletions: inserts need to be coordinated. Note that
    actually collecting the inserts that need to take place is outside the scope of the
    Manager (see the Collector). We do, however, implement a ``sync`` operation that can
    saturates a router with events (dynamically) and sweeps up inserts on session basis
    from an attached collector.
    '''
    def __init__(self, *args, **kwargs):
        '''
        The insert lock is a _reentrant lock_, meaning the same thread can acquire the
        lock again with out deadlocking (simplifying across methods of this class that
        need it).
        '''
        super().__init__(*args, **kwargs)

        self.routers  = []

        self._router    = None
        self._insert_lock = threading.RLock()

    @property
    def router(self):
        if self._router is None:
            self._router = ChainRouter(self.routers)
        return self._router

    def add_router(self, router):
        self.routers.append(router)

    def recreate(
        self,
        schema: Schema[SQLTable],
        engine: SQLEngine
    ) -> None:
        '''
        Ideally this remains open, as we can't necessarily rely on a SQLAlchemy metadata
        object for all kinds of SQLDatabases (would depend on the backend, for instance). 

        Haven't quite nailed down how backend instances should be determined; something
        like SQLAlchemySQLManager doesn't seem great. Nevertheless, this method likely
        cannot be generalized at the "SQL" (general) level.
        '''
        metadata = next(iter(schema._component_set)).obj.metadata
        metadata.drop_all(engine.manager)
        metadata.create_all(engine.manager, checkfirst=True)

    def insert(
        self,
        inserts: list[dict],
        commit=True
    ):
        '''
        '''
        with self._insert_lock:
            res = connection.execute(
                sa.insert(component.obj),
                inserts
            )

            if commit:
                connection.commit()

        return res

    def insert_many(self, connection, inserts: dict):
        '''
        Perform provided table inserts.

            inserts: component-indexed dictionary of insert lists
        '''
        total_inserts = sum([len(ilist) for ilist in inserts.values()])
        if total_inserts < 1: return

        logger.info(f'Total of {total_inserts} sync inserts to perform')

        # TODO: add some exception handling? may be fine w default propagation
        res_list = []
            for component in inserts:
                comp_inserts = inserts[component]
                if len(comp_inserts) == 0: continue

                logger.info(
                    f'Inserting {len(comp_inserts)} out-of-date entries into component "{component}"'
                )

                res = self.insert(connection, component, comp_inserts, commit=False)
                res_list.append(res)

            connection.commit()
            logger.info(f'Insert transaction completed successfully in {time.time()-start:.2f}s')

        return res_list

    def _file_sync_bools(self):
        synced_bools = []
        fpaths = utils.paths.iter_nested_paths(self.collector.basepath, no_dir=True)
        db_rpath_index = self.database.files.group_by('rpath')()
        
        for fpath in fpaths:
            file = File(fpath, self.collector.basepath)
            db_file = db_rpath_index.get(file.rpath)

            file_in_sync = db_file and float(db_file.get('mtime','0')) >= file.mtime
            synced_bools.append(file_in_sync)

        return fpaths, synced_bools

    def sync(
        self,
        limit=0,
        chunk_time=0,
        dry=False,
        chunk_size_cap=1000,
        print_report=True,
    ):
        '''
        Note: 
            Could be dangerous if going from fast file processing to note processing.
            Imagine estimating 1000 iter/sec, then transferring that to the next batch
            when it's more like 0.2 iter/sec. We would lose any chunking. (Luckily, in
            practice, turns out that notes are almost always processed before the huge set
            of nested files lurking and we don't experience this issue.)
        '''
        #filepaths, update_bools = self.collector.file_inserts(dry_run=True)
        #fpaths_to_update = [f for (f, b) in zip(filepaths, update_bools) if not b] 
        filepaths, update_bools = self._file_sync_bools()
        fpaths_to_update = [f for (f, b) in zip(filepaths, update_bools) if not b] 

        def calc_chunk_size(sec_per_file):
            '''
            Chunk cap only applied here; if manually set elsewhere in the method, it's
            intentional e.g., if chunk_time <= 0 and we just want one big iteration.
            '''
            return min(max(int(chunk_time / sec_per_file), 1), chunk_size_cap)
        
        # nothing to do
        if not fpaths_to_update:
            logger.info('Sync has nothing to do, exiting')

        ood_count = len(fpaths_to_update)
        total_fps = len(filepaths) 
        ood_prcnt = ood_count/total_fps*100
        logger.info(
            f'Pre-sync scan yielded {ood_count}/{total_fps} ({ood_prcnt:.2f}%) out-of-date files'
        )

        if limit <= 0: limit = ood_count

        if chunk_time <= 0:
            chunk_size = ood_count
        else:
            chunk_size = calc_chunk_size(5)

        dry_inserts = None
        remaining = limit

        chunk_time_used = 0
        pbar = tqdm(
            desc=f'Adaptive chunked sync for [limit {limit}]',
            total=remaining,
        )

        report = []
        def collect_report():
            report.append({
                'chunk_size': chunk_limit,
                'chunk_time_used': chunk_time_used,
            })

        with pbar as _:
            while remaining > 0:
                time_pcnt = chunk_time_used / max(chunk_time,1) * 100
                pbar.set_description(
                    f'Adaptive chunked sync [size {chunk_size} (max {chunk_size_cap})] '
                    f'[prev chunk {chunk_time_used:.2f}s/{chunk_time}s ({time_pcnt:.2f}%)]'
                )

                chunk_limit = min(remaining, chunk_size)
                start_idx = limit - remaining
                chunk_fpaths = fpaths_to_update[start_idx:start_idx+chunk_limit]

                chunk_time_start = time.time()
                #inserts = self.collector.collect_inserts(
                #    chunk_fpaths,
                #    skip_updated=False, # saves on DB check given provided list
                #)

                # 1) flush synthetic events for the batch through the chained router
                # 2) block until completed and sweep up the collected inserts
                futures = self.router.submit([
                    Event(
                        endpoint=str(self.collector.basepath),
                        name=str(path.relative_to(self.collector.basepath)),
                        action=[0xffffffff], # synthetic action to match any flag filters
                    ) for path in chunk_fpaths
                ])

                for future in tqdm(
                    as_completed(futures),
                    total=chunk_size,
                    desc=f'Awaiting chunk futures [submitted {len(futures)}/{chunk_size}]'
                ):
                    #print('Finished future', cfuture)
                    try:
                        result = future.result()
                    except Exception as e:
                        logger.warning(f"Sync job failed with exception {e}")

                #wait(futures)#, timeout=10)
                inserts = self.collector.collect_inserts()

                chunk_time_used = time.time() - chunk_time_start

                collect_report()
                #if not inserts: break

                if dry:
                    if dry_inserts is None:
                        dry_inserts = inserts
                    else:
                        for ik, iv in inserts.items():
                            dry_inserts[ik].extend(iv)
                else:
                    self.insert(inserts)

                remaining -= chunk_limit

                # re-calculate chunk size
                sec_per_file = chunk_time_used / chunk_size
                chunk_size = calc_chunk_size(sec_per_file)

                #pbar.update(n=limit-remaining)
                pbar.update(n=chunk_limit)
        
        if print_report:
            num_chunks = len(report)
            total_ins  = sum(map(lambda c:c['chunk_size'], report))
            avg_chunk  = total_ins / num_chunks
            total_time = sum(map(lambda c:c['chunk_time_used'], report))
            avg_time   = total_time / num_chunks
            time_match = avg_time / max(chunk_time*100, 1)

            print( 'Sync report:                                              ')
            print(f'    Total chunks         : {num_chunks}                   ')
            print(f'    Total file inserts   : {total_ins} / {limit}          ')
            print(f'    Average chunk size   : {avg_chunk:.2f}                ')
            print(f'    Total time spent     : {total_time:.2f}s              ')
            print(f'    Average time / chunk : {avg_time:.2f}s / {chunk_time}s')
            print(f'    Percent time match   : {time_match:.2f}%              ')
            
        if dry: return dry_inserts

        # create MD object representation current DB state
        pre_metadata = sa.MetaData()
        pre_metadata.reflect(bind=self.engine)

        table_rows = {}
        for table_name, table in pre_metadata.tables.items():
            # pre-migration rows
            select_results = utils.db.sa_execute(self.engine, sa.select(table))
            named_results  = utils.db.named_results(table, select_results)

            table_rows[table_name] = named_results

        #table_update_map = {
        #    'files': (
        #        lambda r: File(Path(NOTEPATH, r['name']), NOTEPATH).get_table_data()
        #    ),
        #    'notes': (
        #        lambda r: Note(Path(NOTEPATH, r['name']), NOTEPATH).get_table_data()
        #    ),
        #    'note_conversion_matter': (
        #    ),
        #}

        # update schema
        self.recreate()

            for table_name, table in tables.metadata.tables.items():
                if table_name not in table_rows: continue

                logger.info(f'Migrating table "{table_name}"')

                if table_name == 'files':
                    update_note_rows = []

                    for i,note_row in enumerate(table_rows['files']):
                        file = File(Path(NOTEPATH, note_row['name']), NOTEPATH)
                        u_note_row = utils.db.prepare_insert(
                            tables.files,
                            file.get_table_data()
                        )
                        note_row.update({k:v for k,v in u_note_row.items() if v})
                        update_note_rows.append(note_row)

                    table_rows['files'] = update_note_rows
                elif table_name == 'notes':

                    for note_row in table_rows['notes']:
                        note = Note(Path(NOTEPATH, note_row['name']), NOTEPATH)
                            tables.notes,
                            note.get_table_data()
                        )

                    table_rows['notes'] = update_note_rows

                # generic update blueprint for later; now doesn't matter as conversion based
                # inserts require re-converting...so you may as well just do a full re-build.
                #update_rows = []
                #if table_name in table_update_map:
                #    for i,row in enumerate(table_rows['files']):
                #        update_obj = table_update_map[table_name](row)

                #        u_row = utils.db.prepare_insert(
                #            table,
                #            note.get_table_data()
                #        )
                #        note_row.update({k:v for k,v in u_note_row.items() if v})
                #        update_note_rows.append(note_row)

                connection.execute(
                    sa.insert(table),
                    table_rows[table_name]
                )

import numpy as np

class VSSManager(Manager):

        from localsys.db.databases.core import CoreDatabase

        self.core_db = CoreDatabase()

        chunks    = []
        chunk_ids = []
        block_map = {}
        for note_name, note_groupby in self.core_db.blocks().items():
            blocks = note_groupby.get('aggregates', [])
            for block in blocks:
                if block.get('format') != 'src': continue

                block_name = block.get('name', '')
                block_content = block.get('content', '')
                block_map[block_name] = block_content

                block_chunks, chunk_ranges = utils.embed.split_block(block_content)

                chunks.extend(block_chunks)
                chunk_ids.extend([f'{block_name}@{r[0]}:{r[1]}' for r in chunk_ranges])

        logger.info(f'VSS pre-index SELECT took {time.time()-start:.2f}s')
    
        # one-time batched embedding
        chunk_embeddings = self.database.access.embed_chunks(chunks)

        logger.info(f'VSS embeddings took {time.time()-start:.2f}s')

        # aggregate embeddings up the content hierarchy
        zero_embed = lambda: np.atleast_2d(
            np.zeros(self.database.access._embedding_size, dtype='float32')
        )
        block_embedding_map = defaultdict(zero_embed)
        note_embedding_map  = defaultdict(zero_embed)

        for i, chunk_id in enumerate(chunk_ids):
            at_split   = chunk_id.split('@')
            note_name  = at_split[0]
            block_name = '@'.join(at_split[:3])

            block_embedding_map[block_name] += chunk_embeddings[i]
            note_embedding_map[note_name]   += chunk_embeddings[i]

        # re-normalize aggregated embeddings
        # > little inefficient but gobbled up asymptotically by chunk processing
        # > could o/w first convert each map to a full 2D np array, then back
        for embed_map in [block_embedding_map, note_embedding_map]:
            for k, agg_embedding in embed_map.items():
                embed_map[k] = utils.embed.unit_row_wise(agg_embedding)

        logger.info(f'VSS recreate: pre-index SELECT took {time.time()-start:.2f}s')

        # cols = ['rowid', 'chunk_embedding']
        # def pack_inserts(embeddings):
        #     return [
        #         { cols[0]: i, cols[1]: embedding.tobytes() }
        #         for i, embedding in enumerate(embeddings)
        #     ]

        # chunk_inserts = pack_inserts(chunk_embeddings)
        # block_inserts = pack_inserts(block_embedding_map.values())
        # note_inserts  = pack_inserts(note_embedding_map.values())
            
        chunk_ids = chunk_ids
        block_ids = list(block_embedding_map.keys())
        note_ids  = list(note_embedding_map.keys())

        block_embeddings = np.vstack(list(block_embedding_map.values()))
        note_embeddings  = np.vstack(list(note_embedding_map.values()))

        blocks = [block_map[b] for b in block_ids]
        notes  = note_ids

        self.database.access.write_embeddings({
            'chunks': (chunk_ids, chunk_embeddings, chunks),
            'blocks': (block_ids, block_embeddings, blocks),
            'notes' : (note_ids,  note_embeddings,  notes),
        })

        logger.info(f'Post-embedding parsing took {time.time()-start:.2f}s')

        # utils.db.create_vss0(
        #     self.engine,
        #     'chunks',
        #     columns=list(cols),
        #     inserts=chunk_inserts,
        #     populate=True,
        #     reset=True,
        #     embedding_size=self._embedding_size,
        # )

    def _archive_note_embed(self):
        for block in blocks:
            content = block.get('content')
            if not content: continue

            block_chunks = utils.embed.split_block(content)
            chunks.extend(block_chunks)
            
        embeddings = utils.embed.embed_chunks(chunks)
        block_embeddings = utils.embed.map_groups(embeddings, embed_block_map)

        self.convert_map['vect'] = block_embeddings

        note_embedding = utils.embed.unit_row_wise(
            embeddings.sum(axis=0).reshape(1, -1)
        )

        return {
            'content': note_embedding.tobytes(),
        }

'''
Mapper

Example:

    mapper = Mapper[sa.Table]()
    
    mapper.attach(
        Type,
        attr_comp=TypeTable,
        coll_comp=CollateTable,
        coll_groups={
            'name': NameConversions
        }
    )

.. admonition:: Development log

    - Overruled design decision: Mappers were previously designed to map from a specific
      CO3 hierarchy to a specific Schema. The intention was to allow only related types to
      be attached to a single schema, at least under a particular Mapper. The type
      restriction has since been removed, however, as it isn't particularly well-founded.
      During ``collect()``, a particular instance collects data from both its attributes and
      its collation actions. It then repeats the same upward for parent types (part of the
      same type hierarchy), and down to components (often not part of the same type
      hierarchy). As such, to fully collect from a type, the Mapper needs to leave
      registration open to various types, not just those part of the same hierarchy.
'''
from typing import Callable, Any

from co3.co3        import CO3
from co3.schema     import Schema
from co3.collector  import Collector
from co3.component  import Component
from co3.components import ComposableComponent

class Mapper[C: Component]:
    '''
    Mapper base class for housing schema components and managing relationships between CO3
    types and storage components (of type C).

    Mappers are responsible for two primary tasks:

    1. Attaching CO3 types to database Components from within a single schema
    2. Facilitating collection of Component-related insertion data from instances of
       attached CO3 types

    Additionally, the Mapper manages its own Collector and Composer instances. The
    Collector receives the inserts from ``.collect()`` calls, and will subsequently be
    "dropped off" at an appropriate Database's Manager to actually perform the requested
    inserts (hence why we tie Mappers to Schemas one-to-one). 

    .. admonition:: Dev note
        the Composer needs reconsideration, or at least its positioning directly in this
        class. It may be more appropriate to have at the Schema level, or even just
        dissolved altogether if arbitrary named Components can be attached to schemas.

        - Consider pushing this into a Mapper factory; on init, could check if provided
          Schema wraps up composable Components or not
    '''
    _collector_cls: type[Collector[C]] = Collector[C]

        '''
            schema: Schema object holding the set of components eligible as attachment
                    targets for registered CO3 types
        '''

        self.collector = self._collector_cls(schema)

        self.attribute_comps:  dict[type[CO3], C] = {}
        self.collation_groups: dict[type[CO3], dict[str|None, C]] = defaultdict(dict)

    def _check_component(self, comp: str | C):
        if type(comp) is str:
            comp_key = comp
            comp = self.schema.get_component(comp_key)
            if comp is None:
                raise ValueError(
                    f'Component key {comp_key} not available in attached schema'
                )
        else:
            if comp not in self.schema:
                raise TypeError(
                    f'Component {comp} not registered to Mapper schema {self.schema}'
                )

        return comp

    def attach(
        self,
        type_ref    : type[CO3],
        attr_comp   : str | C,
        coll_comp   : str | C | None                   = None,
        coll_groups : dict[str | None, str | C] | None = None,
    ) -> None:
        '''
            type_ref:    CO3 subtype to map to provided storage components
            attr_comp:   storage component for provided type's canonical attributes
            coll_comp:   storage component for provided type's default/unnamed collation
                         targets
            coll_groups: storage components for named collation groups; dict mapping group
                         names to components
        '''
        # check attribute component in registered schema
        attr_comp = self._check_component(attr_comp)
        self.attribute_comps[type_ref] = attr_comp

        # check default component in registered schema
        if coll_comp is not None:
            coll_comp = self._check_component(coll_comp)
            self.collation_groups[type_ref][None] = coll_comp

        # check if any component in group dict not in registered schema
        if coll_groups is not None:
            for coll_key in coll_groups:
                coll_groups[coll_key] = self._check_component(coll_groups[coll_key])

            self.collation_groups[type_ref].update(coll_groups)

    def attach_many(
        self,
        type_list: list[type[CO3]],
        attr_name_map: Callable[[type[CO3]], str | C],
        coll_name_map: Callable[[type[CO3], str], str | C] | None = None,
    ) -> None:
        '''
        Auto-register a set of types to the Mapper's attached Schema. Associations are
        made from types to both attribute and collation component names, through
        ``attr_name_map`` and ``coll_name_map``, respectively. Collation targets are inferred
        through the registered groups in each type.

            type_ref:      reference to CO3 type
            attr_name_map: function mapping from types/classes to attribute component names
                           in the attached Mapper Schema
            coll_name_map: function mapping from types/classes & action groups to
                           collation component names in the attached Mapper Schema. ``None``
                           is passed as the action group to retrieve the default
                           collection target.
        '''
        for _type in type_list:
            attr_comp = attr_name_map(_type)
            coll_groups = {}

            if coll_name_map:
                for action_group in _type.group_registry:
                    coll_groups[action_group] = coll_name_map(_type, action_group)

            self.attach(_type, attr_comp, coll_groups=coll_groups)

    def get_attr_comp(
        self,
        co3_ref: CO3 | type[CO3]
    ) -> C | None:
        type_ref = co3_ref
        if isinstance(co3_ref, CO3):
            type_ref = co3_ref.__class__

        return self.attribute_comps.get(type_ref, None)

    def get_coll_comp(
        self,
        co3_ref: CO3 | type[CO3],
        group=str | None,

        return self.collation_groups.get(type_ref, {}).get(group, None)

    def collect(
        self,
        obj           : CO3,
        action_keys   : list[str] = None,
        action_groups : list[str] = None,
    ) -> list:
        '''
        Stages inserts up the inheritance chain, and down through components.

        Note:
            Even with ORM, a method like this would be needed to trace up parent tables and
            how inserts should be handled for inheritance. ORM would make component
            inserts a little easier perhaps, since they can be attached as attributes to
            constructed row objects and a sa.Relationship will handle the rest. Granted,
            we don't do a whole lot more here: we just call ``collect`` over those
            components, adding them to the collector session all the same.

            obj: CO3 instance to collect from
            action_keys: keys for actions to collect from
            action_group: action group names to run all actions for

        Returns: dict with keys and values relevant for associated SQLite tables
        '''
        # default is to have no actions
        if action_keys is None:
            action_keys = []
            #action_keys = list(obj.action_registry.keys())

        receipts = []
        for _cls in reversed(obj.__class__.__mro__[:-2]):
            attribute_component = self.get_attr_comp(_cls)

            # require an attribute component for type consideration
            if attribute_component is None:
                continue

            self.collector.add_insert(
                attribute_component,
                obj.attributes,
                receipts=receipts,
            )

            for action_key in action_keys:
                collation_data = obj.collate(action_key)

                # if method either returned no data or isn't registered, ignore
                if collation_data is None:
                    continue

                _, action_groups = obj.action_registry.get(action_key, (None, []))
                    collation_component = self.get_coll_comp(_cls, group=action_group)

                    if collation_component is None:
                        continue

                    # gather connective data for collation components
                    connective_data = obj.collation_attributes(action_key, action_group)

                    self.collector.add_insert(
                        collation_component,
                        {
                            **connective_data,
                            **collation_data, 
                        },
                        receipts=receipts,
                    )

        # handle components
        for comp in [c for c in obj.components if isinstance(c, CO3)]:
            receipts.extend(comp.collect(collector, formats=formats))

        return receipts

class ComposableMapper[C: ComposableComponent](Mapper[C]):
    '''
    .. admonition:: class design

        Heavily debating between multiple possible design approaches here. The main
        purpose of this subtype is make clear the need for additional compositional
        mapping details, namely functions that can produce pairwise join conditions for
        both the attribute tree (vertical traversal) and the collation components
        (horizontal traversal). Here's a few remarks:

        - I want the necessary maps to provided/stored *outside* of ``compose`` calls to
          reduce overhead for downstream callers. It's awkward to have think about the
          exact attr-to-attr associations each time you want a type's associated
          composition, especially when they don't change under the same Mapper (i.e.,
          if you have the Mapper reference, the compositional associations should be
          implicit).
        - The barebones spec here appears to be two pairwise "composer" maps: one for
          attribute comps, and one for collation comps. For now I think this makes sense
          as additional init params, but there may later be reason to wrap this up a bit
          more.
        - Considering the full deprecation for the Composer type, or whether this could be
          the place where it serves some purpose. Aesthetically, there's symmetry with the
          ``collect`` and Collector method-type pairing, but that isn't a good enough reason
          to justify a separate type here. The difference is that Collector instances
          actually store type references, whereas the considered Composer type would
          effectively just be a convenient collection of utility functions. Still possibly
          useful, but not as clearly justifiable.
        - If a separate Composer type were to be justified here, it would serve as a
          "reusable connective tissue" for possibly many Mappers with the same kinds of
          edge-wise relationships. Can think of it like this:

          * Schemas collect up "nodes" (Components). These are explicit storage structures
            in a DB, and can include some explicit attribute connections (foreign keys),
            although those are connections made on the DB side.
          * Mappers provide an exoskeleton for a Schema's nodes. It structures Components into
            attributes and collation types, and additionally ties them to external CO3
            types. The handy analogy here has been that attribute comps connect
            *vertically* (in a tree like fashion; point up for parents and down for
            children), and collation comps point *horizontally* (or perhaps more aptly,
            *outward*; at each node in the attribute tree, you have a "circle" of
            collation comps that can point to it, and are not involved as formal tree
            nodes. Can maybe think of these like "ornaments" or bulbs or orbitals).
          * While the Mappers may provide the "bones," there's no way to communicate
            *across* them. While I might know that one attribute is the "parent" of
            another, I don't know *why* that relationship is there. A Composer, or the
            composer details to be provided to this class, serve as the "nerves" to be
            paired with the bone, actually establishing a line of communication. More
            specifically, the nerves here are attribute-based mappings between pairs of
            Components, i.e., (generalized) join conditions.

        - Note that, by the above logic, we should then want/need a type to manage the
          functions provided to ``attach_many``. These functions help automatically
          characterize the shape of the type skeleton in the same way the proposed
          Composer wrapper would. In fact, the barebones presentation here is really just
          the same two function signatures as are expected by that method. The above
          analogy simply made me ask why the "bones" wouldn't be reusable if the "nerves"
          were going to be. So we should perhaps coordinate a decision on this front; if
          one goes, the other must as well. This may also help me keep it simpler for the
          time being.
        - One other aspect of a dedicated Composer type (and by the above point, a
          hypothetical type to aid in ``attach_many`` specification) could have some sort of
          "auto" feature about it. With a clear enough "discovery system," we could
          encourage certain kinds of Schemas and components are named and structured. Such
          an auto-composer could "scan" all components in a provided Schema and attempt to
          find common attributes across tables that are unlinked (i.e., the reused
          column names implicit across types in the attribute hierarchy; e.g., File.name
          -> Note.name), as well as explicit connections which may suggest collation
          attachment (e.g., ``note_conversions.name --FK-> Note.name``). This, of course,
          could always be overridden with manual specification, but being aware of some
          automatic discovery structures could help constrain schema definitions to be
          more in-line with the CO3 operational model. That all being said, this is a
          large amount of complexity and should likely be avoided until necessary.
    '''
        self,
        schema           : Schema[C],
        attr_compose_map : Callable[[str | C, str | C], Any] | None = None,
        coll_compose_map : Callable[[str | C, str | C], Any] | None = None,
    ):
        super().__init__(schema)

        self.attr_compose_map = attr_compose_map
        self.coll_compose_map = coll_compose_map

        self,
        co3_ref:       CO3 | type[CO3],
        action_groups: list[str] | None = None,
        *compose_args,
        **compose_kwargs,
    ):
        '''
        Compose tables up the type hierarchy, and across through action groups to
        collation components.

        Note:
            Comparing to ORM, this method would likely also still be needed, since it may
            not be explicitly clear how some JOINs should be handled up the inheritance
            chain (for components / sa.Relationships, it's a little easier).

            obj: either a CO3 instance or a type reference
        '''

        attr_comp_agg = None
        for _cls in reversed(type_ref.__mro__[:-2]):
            attr_comp = self.get_attr_comp(_cls)

            if attr_comp is None:
                continue

            # compose horizontally with components from provided action groups
            coll_comp_agg = attr_comp
            if action_groups is not None:
                    coll_comp = self.get_coll_comp(_cls, group=action_group)

                    if coll_comp is None:
                        continue

                    # note how the join condition is specified using the non-composite
                    # `attr_comp` and new `coll_comp`; the composite doesn't typically
                    # have the same attribute access and needs a ref to a specific comp
                    compose_condition = self.coll_compose_map(attr_comp, coll_comp)

                    coll_comp_agg = coll_comp_agg.compose(
                        coll_comp,
                        compose_condition,
                        *compose_args,
                        **compose_kwargs,
                    )

            if attr_comp_agg is None:
                attr_comp_agg = coll_comp_agg
            else:
                # note the reduced attr_comp ref passed to compose map, rather than
                # coll_comp_agg produced above; this is provided as the compose comp, though
                compose_condition = self.attr_compose_map(attr_comp_agg, attr_comp)
                attr_comp_agg = attr_comp_agg.compose(
                    coll_comp_agg,
                    compose_condition,
                    *compose_args,
                    **compose_kwargs,
                )

        return attr_comp_agg

'''
Schema

Schema objects are used to:

- Semantically group related storage components
- Tell databases what components to create/remove together
- Provide target contexts for connected CO3 type systems within Mappers

'''

class Schema[C: Component]:
    def __init__(self):
        self._component_set = set()
        self._component_map = {}

    def __contains__(self, component: C):
        return component in self._component_set

    def add_component(self, component: C):
        self._component_set.add(component)
        self._component_map[component.name] = component

    def get_component(self, name: str):
        return self._component_map.get(name)

class RelationalSchema[R: Relation](Schema[R]):
    pass

class SQLSchema(RelationalSchema[SQLTable]):
    def from_metadata(cls, metadata: sa.MetaData):
        instance = cls()

        for table in metadata.tables.values():
            comp = SQLTable.from_table(table)
            instance.add_component(comp)

        return instance
        
'''
Example usage for this file's utilities:

    # get SA engine, creating folder hierarchy to provided DB path
    engine = db.get_engine(<path>)
    
    # execute a single SA statement, returns a CursorResult
    select_results = db.sa_execute(engine, sa.select(<table>))
    
    # convert raw results to dictionaries, keys corresponding to col names
    select_dicts = db.named_results(<table>, select_results)
    
    # use table defaults and cols to create compliant insert
    insert_dicts = [ db.prepare_insert(<table>, sd) for sd in select_dicts ]
    
    # perform a bulk insert
    with engine.connect() as connection:
        connection.execute(
            sa.insert(<table>),
            insert_dicts
        )
'''

import functools

def get_engine(db_path, echo=False):
    Path(db_path).parent.mkdir(parents=True, exist_ok=True)
    return sa.create_engine(f"sqlite:///{db_path}", echo=echo)

def named_results(table, results):
    '''
    Note the implications of this for results from compound tables containing the same
    column names: only the last column name will be indexed.
    '''
    return [
        { c.name:r[i] for i, c in enumerate(table.columns) }
        for r in results
    ]

def deferred_fkey(target, **kwargs):
    return sa.ForeignKey(
        target,
        deferrable=True,
        initially='DEFERRED',
        **kwargs
    )

def deferred_cd_fkey(target, **kwargs):
    '''
    Prefer this when using FKEYs; need to really justify *not* having a CASCADE deletion
    enabled
    '''
    return deferred_fkey(target, ondelete='CASCADE', **kwargs)

def get_column_names_str_table(engine, table: str):
    col_sql = f'PRAGMA table_info({table});'
        try:
            cols = connection.execute(sa.text(col_sql))
        except sa.exc.OperationalError as e:
            logger.error(f'Column retrieval for table "{table}" failed')
            raise

    return cols

def create_fts5(
        engine,
        table: sa.Table | str,
        columns=None,
        populate=False,
        inserts=None,
        reset_fts=False,
        tokenizer='unicode61',
    ):
    '''
    Create and optionally populate an FTS5 table in SQLite. Can be used directly for
    existing tables in the same database. It can also be used for composite tables (i.e.,
    those created from JOINs) or really any other data by providing explicit inserts and
    column names to use during population.

        table: either SQLAlchemy table instance, or table name string
        columns: list of SQLAlchemy table columns to insert into virtual table. These
                 columns must be present in the provided table if not manually specifying
                 inserts (since the table must be queried automatically)
        inserts: 
    '''
    is_sa_table = isinstance(table, sa.Table)
    table_name  = table.name if is_sa_table else table

    if columns is None:
        if is_sa_table:
            columns = [c.name for c in table.c] 
        else:
            columns = get_column_names_str_table(engine, table)

    col_str = ", ".join(columns)
    fts_table_name = f'{table_name}_fts_{tokenizer}'

    sql = f"""
    CREATE VIRTUAL TABLE {fts_table_name} USING fts5
    (
        {col_str},
        tokenize = '{tokenizer}'
    );
    """

    sql_insert = f"""
    INSERT INTO {fts_table_name}
    (
        {col_str}
    )
    """
    if inserts is None:
        sql_insert += f"""
            SELECT {col_str}
            FROM {table_name};
        """
    else:
        VALUES ({', '.join(':' + c for c in columns)})
        """

    sql_drop = f"DROP TABLE IF EXISTS {fts_table_name}"

        if reset_fts:
            connection.execute(sa.text(sql_drop))

        connection.execute(sa.text(sql))

        if populate:
            if inserts is None:
                connection.execute(sa.text(sql_insert))
            else:
                    sa.text(sql_insert),
                    inserts,
                )

        connection.commit()

def populate_fts5(engine, tables, columns=None, inserts=None):
    # create indexes
    tokenizers = ['unicode61', 'porter', 'trigram']

    for table in tables:
        for tokenizer in tokenizers:
            start = time.time()
            create_fts5(
                engine,
                table,
                columns=columns,
                populate=True,
                inserts=inserts,
                reset_fts=True,
                tokenizer=tokenizer
            )

            is_sa_table = isinstance(table, sa.Table)
            table_name  = table.name if is_sa_table else table
            print(f'Created FTS5 index for table "{table_name}+{tokenizer}"; took {time.time() - start}s')

def create_vss0(
        engine,
        reset=False,
        embedding_size=384,
    ):
    '''
    Create a VSS table.

        inserts: 
    '''

        else:

    vss_table_name = f'{table_name}_vss'

    sql = f"""
    CREATE VIRTUAL TABLE {vss_table_name} USING vss0
    (
        chunk_embedding({embedding_size}),
        query_embedding({embedding_size}),
    );
    """

    INSERT INTO {vss_table_name}
    (
        rowid, chunk_embedding
    )
    """
        """
    else:
        """

    sql_drop = f"DROP TABLE IF EXISTS {vss_table_name}"

        if reset:

            else:
                    inserts,
                )

def fts5_prep_composite(engine, table, table_name, columns=None):
    '''
    Helper method for prepping JOIN tables for FTS5 creation.
    '''
    table.name = table_name

    rows, cols = utils.db.sa_execute(
        engine,
        sa.select(*select_cols).select_from(all_search),
        include_cols=True
    )
    rows = utils.db.result_dicts(rows)

    return table, rows, cols

import re

camel2snake_regex = re.compile(r'(?<!^)(?=[A-Z])')

def camel_to_snake(text):
    return camel2snake_regex.sub('_', text).lower()