Improvements to Relationships, updated documentation

docs/1_getting_started.md

@@ -26,9 +26,66 @@ from typedal.for_py4web import TypeDAL
 db = TypeDAL("sqlite:memory")
 ```
 
-TypeDAL accepts the same connection string format and other arguments as `pydal.DAL`. Again,
-see [their documentation](http://www.web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#The-DAL-A-quick-tour)
-for more info about this.
+TypeDAL accepts the same connection string format and other arguments as `pydal.DAL`.
+Again, see
+[their documentation](http://www.web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#The-DAL-A-quick-tour)
+for more info about this. For additional configuration options specific to TypeDAL, see the [7. Advanced Configuration](./7_configuration.md) page.
 
 When using py4web, it is recommended to import the py4web-specific TypeDAL, which is a Fixture that handles database
-connections on request (just like the py4web specific DAL class does).
+connections on request (just like the py4web specific DAL class does). More information about this can be found on [5. py4web](./5_py4web.md)
+
+### Simple Queries
+
+For direct SQL access, use `executesql()`:
+
+```python
+rows = db.executesql("SELECT * FROM some_table")
+```
+
+#### Safely Injecting Variables
+
+Use t-strings (Python 3.14+) for automatic SQL escaping:
+
+```python
+name = "Robert'); DROP TABLE Students;--"
+rows = db.executesql(t"SELECT * FROM some_table WHERE name = {name}")
+```
+
+Or use the `placeholders` argument with positional or named parameters:
+
+```python
+# Positional
+rows = db.executesql(
+    "SELECT * FROM some_table WHERE name = %s AND age > %s",
+    placeholders=[name, 18]
+)
+
+# Named
+rows = db.executesql(
+    "SELECT * FROM some_table WHERE name = %(name)s AND age > %(age)s",
+    placeholders={"name": name, "age": 18}
+)
+```
+
+#### Result Formatting
+
+By default, `executesql()` returns rows as tuples. To map results to specific fields, use `fields` (takes
+Field/TypedField objects) or `colnames` (takes column name strings):
+
+```python
+rows = db.executesql(
+    "SELECT id, name FROM some_table",
+    colnames=["id", "name"]
+)
+
+rows = db.executesql(
+    "SELECT id, name FROM some_table",
+    fields=[some_table.id, some_table.name]  # Requires table definition
+)
+```
+
+You can also use `as_dict` or `as_ordered_dict` to return dictionaries instead of tuples.
+
+Most of the time, you probably don't want to write raw queries. For that, you'll need to define some tables!
+Head to [2. Defining Tables](./2_defining_tables.md) to learn how.
+

docs/2_defining_tables.md

@@ -102,5 +102,6 @@ def my_after_delete(query: Set):
 
 row.delete_record() # to trigger
 MyTable.where(...).delete() # to trigger
-
 ```
+
+"Now that we have some tables, it's time to actually query them! Let's go to [3. Building Queries](./3_building_queries.md) to learn how.

docs/3_building_queries.md

@@ -30,55 +30,115 @@ The query builder uses the builder pattern, so you can keep adding to it (in any
 data:
 
 ```python
-Person
+builder = Person
 .where(Person.id > 0)
-.where(Person.id < 99, Person.id == 101)
+.where(Person.id < 99, Person.id == 101)  # id < 99 OR id == 101
 .select(Reference.id, Reference.title)
 .join('reference')
 .select(Person.ALL)
-.paginate(limit=5, page=2)  # final step: actually runs the query
+
+# final step: actually runs the query:
+builder.paginate(limit=5, page=2)
+
+# to get the SQL (for debugging or subselects), you can use:
+builder.to_sql()
 ```
 
 ```sql
-SELECT "person".*,
-       "reference"."id",
-       "reference"."title"
-FROM "person",
-     "reference"
-WHERE (("person"."id" IN (SELECT "person"."id"
-                          FROM "person"
-                          WHERE ((("person"."id" > 0)) AND
-                                 (("person"."id" < 99) OR ("person"."id" = 101)))
-                          ORDER BY "person"."id" LIMIT 1 OFFSET 0))
-  AND ("person"."reference" = "reference"."id") );
+SELECT "person".*
+     , "reference"."id"
+     , "reference"."title"
+    FROM "person"
+       , "reference"
+    WHERE (("person"."id" IN (SELECT "person"."id"
+                                  FROM "person"
+                                  WHERE ((("person"."id" > 0)) AND
+                                         (("person"."id" < 99) OR ("person"."id" = 101)))
+                                  ORDER BY "person"."id"
+                                  LIMIT 1 OFFSET 0))
+        AND ("person"."reference" = "reference"."id"));
 ```
 
 ### where
 
 In pydal, this is the part that would be in `db(...)`.
 Can be used in multiple ways:
 
-- `.where(Query)` -> with a direct query such as `Table.id == 5`
+- `.where(query)` -> with a direct query such as `query = (Table.id == 5)`
 - `.where(lambda table: table.id == 5)` -> with a query via a lambda
 - `.where(id=5)` -> via keyword arguments
+- `.where({"id": 5})` -> via a dictionary (equivalent to keyword args)
+- `.where(Table.field)` -> with a Field directly, checks if field is not null
+
+When using multiple `.where()` calls, they will be ANDed together:
+`.where(lambda table: table.id == 5).where(active=True)` equals `(table.id == 5) & (table.active == True)`
 
-When using multiple where's, they will be ANDed:  
-`.where(lambda table: table.id == 5).where(lambda table: table.id == 6)` equals `(table.id == 5) & (table.id=6)`  
-When passing multiple queries to a single .where, they will be ORed:  
-`.where(lambda table: table.id == 5, lambda table: table.id == 6)` equals `(table.id == 5) | (table.id=6)`
+When passing multiple arguments to a single `.where()`, they will be ORed:
+`.where({"id": 5}, {"id": 6})` equals `(table.id == 5) | (table.id == 6)`
 
 ### select
 
-Here you can enter any number of fields as arguments: database columns by name ('id'), by field reference (table.id) or
-other (e.g. table.ALL).
+Here you can enter any number of fields as arguments: database columns by name ('id'), by field reference (Table.id),
+other (e.g. Table.ALL), or Expression objects.
 
 ```python
 Person.select('id', Person.name, Person.ALL)  # defaults to Person.ALL if select is omitted.
 ```
 
-You can also specify extra options such as `orderby` here. For supported keyword arguments, see
+You can also specify extra options as keyword arguments. Supported options are: `orderby`, `groupby`, `limitby`,
+`distinct`, `having`, `orderby_on_limitby`, `join`, `left`, `cache`, see also
 the [web2py docs](http://www.web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#orderby-groupby-limitby-distinct-having-orderby_on_limitby-join-left-cache).
 
+```python
+Person.select(Person.name, distinct=True)
+```
+
+If you only want a list of name strings (in this example) instead of Person instances, you could use the column() method
+instead:
+
+```python
+Person.column(Person.name, distinct=True)
+```
+
+You can use `.orderby(*fields)` as an alternative to `select(orderby=...)`:
+
+```python
+Person.where(...).orderby(~Person.name, "age")
+```
+
+`.orderby()` accepts field references (`Table.field` or `"field_name""`), reverse ordering (`~Table.field`), or the
+literal `"<random>"`. Multiple field references can be passed (except when using `<random>`).
+
+#### Raw SQL Expressions
+
+For complex SQL that can't be expressed with field references, use `sql_expression()`:
+
+```python
+# Simple arithmetic
+expr = db.sql_expression("age * 2")
+Person.select(expr)
+
+# Safe parameter injection with t-strings (Python 3.14+)
+min_age = 21
+expr = db.sql_expression(t"age >= {min_age}", output_type="boolean")
+Person.where(expr).select()
+
+# Positional arguments
+expr = db.sql_expression("age > %s AND status = %s", 18, "active", output_type="boolean")
+Person.where(expr).select()
+
+# Named arguments
+expr = db.sql_expression(
+    "EXTRACT(year FROM %(date_col)s) = %(year)s",
+    date_col="created_at",
+    year=2023,
+    output_type="boolean"
+)
+Person.where(expr).select()
+```
+
+Expressions can be used in `where()`, `select()`, `orderby()`, and other query methods.
+
 ### join
 
 Include relationship fields in the result.
@@ -93,6 +153,8 @@ This can be overwritten with the `method` keyword argument (left or inner)
 Person.join('articles', method='inner')  # will only yield persons that have related articles
 ```
 
+For more details about relationships and joins, see [4. Relationships](./4_relationships.md).
+
 ### cache
 
 ```python
@@ -120,7 +182,7 @@ time.
 class ...
 ```
 
-In order to enable this functionality, TypeDAL adds a `before update` and `before delete` hook to your tables, 
+In order to enable this functionality, TypeDAL adds a `before update` and `before delete` hook to your tables,
 which manages the dependencies. You can disable this behavior by passing `cache_dependency=False` to `db.define`.
 Be aware doing this might break some caching functionality!
 
@@ -136,12 +198,15 @@ The Query Builder has a few operations that don't return a new builder instance:
 - paginate: this works similarly to `collect`, but returns a PaginatedRows instead, which has a `.next()`
   and `.previous()` method to easily load more pages.
 - collect_or_fail: where `collect` may return an empty result, this variant will raise an error if there are no results.
+- execute: get the raw rows matching your query as returned by pydal, without entity mapping or relationship loading.
+  Useful for subqueries or when you need lower-level control.
 - first: get the first entity matching your query, possibly with relationships loaded (if .join was used)
 - first_or_fail: where `first` may return an empty result, this variant will raise an error if there are no results.
+- to_sql: get the SQL string that would run, useful for debugging, subqueries and other advanced SQL operations.
 - update: instead of selecting rows, update those matching the current query (see [Delete](#delete))
 - delete: instead of selecting rows, delete those matching the current query (see [Update](#update))
 
-Additionally, you can directly call `.all()`, `.collect()`, `.count()`, `.first()` on a model.
+Additionally, you can directly call `.all()`, `.collect()`, `.count()`, `.first()` on a model (e.g. `User.all()`).
 
 ## Update
 

docs/4_relationships.md

@@ -20,33 +20,44 @@ class Post(TypedTable):
     author: Author
 
 
-authors_with_roles = Author.join('role').collect()
+authors_with_roles = Author.join('roles').collect()
 posts_with_author = Post.join().collect()  # join can be called without arguments to join all relationships (in this case only 'author')
+post_deep = Post.join("author.roles").collect()  # nested relationship, accessible via post.author.roles
 ```
 
 In this example, the `Post` table contains a reference to the `Author` table. In that case, the `Author` `id` is stored
-in the
-`Post`'s `author` column.
+in the `Post`'s `author` column.
 Furthermore, the `Author` table contains a `list:reference` to `list[Role]`. This means multiple `id`s from the `Role`
-table
-can be stored in the `roles` column of `Author`.
+table can be stored in the `roles` column of `Author`.
 
 For these two cases, a Relationship is set-up automatically, which means `.join()` can work with those.
 
+### Alternative Join Syntax
+
+You can pass the relationship object directly instead of its name as a string:
+
+```python
+posts_with_author = Post.join(Post.author).collect()
+```
+
+This works, but note that `Post.author` is typed as `Relationship[Author]` at the class level, while `row.author` is
+typed as `Author` at the instance level. Some editors may complain about type mismatches when using this syntax (e.g.,
+reporting that `list[Tag]` isn't a `Relationship`). If you encounter type checking issues, use the string syntax
+instead.
+
 ## Other Relationships
 
 To get the reverse relationship, you'll have to tell TypeDAL how the two tables relate to each other (since guessing is
 complex and unreliable).
 
-For example, to set up the reverse relationshp from author to posts:
+For example, to set up the reverse relationship from author to posts:
 
 ```python
 @db.define()
 class Author(TypedTable):
     name: str
 
     posts = relationship(list["Post"], condition=lambda author, post: author.id == post.author, join="left")
-
 ```
 
 Note that `"Post"` is in quotes. This is because the `Post` class is defined later, so a reference to it is not
@@ -62,7 +73,7 @@ class Role(TypedTable):
     authors = relationship(list["Author"], condition=lambda role, author: author.roles.contains(role.id), join="left")
 ```
 
-Here, contains is used since `Author.roles` is a `list:reference`.
+Here, `contains` is used since `Author.roles` is a `list:reference`.
 See [the web2py docs](http://www.web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#list-type-and-contains)
 for more details.
 
@@ -82,8 +93,8 @@ class Sidekick(TypedTable):
     superhero: SuperHero
 ```
 
-In this example, `Relationship["Sidekick"]` is added as an extra type hint, since the reference to the table
-in `relationship("Sidekick", ...)` is a string. This has to be passed as a string, since the Sidekick class is defined
+In this example, `Relationship["Sidekick"]` is added as an extra type hint, since the reference to the table in
+`relationship("Sidekick", ...)` is a string. This has to be passed as a string, since the Sidekick class is defined
 after the superhero class.
 Adding the `Relationship["Sidekick"]` hint is optional, but recommended to improve editor support.
 
@@ -107,6 +118,7 @@ class Post(TypedTable):
         tag.on(tag.id == tagged.tag),
     ])
 
+
 # without unique alias:
 
 @db.define()
@@ -126,7 +138,65 @@ class Tagged(TypedTable):
 ```
 
 Instead of a `condition`, it is recommended to define an `on`. Using a condition is possible, but could lead to pydal
-generation a `CROSS JOIN` instead of a `LEFT JOIN`, which is bad for performance.
+generating a `CROSS JOIN` instead of a `LEFT JOIN`, which is bad for performance.
 In this example, `Tag` is connected to `Post` and vice versa via the `Tagged` table.
 It is recommended to use the tables received as arguments from the lambda (e.g. `tag.on` instead of `Tag.on` directly),
 since these use aliases under the hood, which prevents conflicts when joining the same table multiple times.
+
+## Lazy Loading and Explicit Relationships
+
+### Lazy Policy
+
+The `lazy` parameter on a relationship controls what happens when you access relationship data without explicitly
+joining it first:
+
+```python
+@db.define()
+class User(TypedTable):
+    name: str
+    posts = relationship(list["Post"], condition=lambda user, post: user.id == post.author, lazy="forbid")
+```
+
+Available policies:
+
+- **`"forbid"`**: Raises an error. Prevents N+1 query problems by making them fail fast.
+- **`"warn"`**: Returns an empty value (empty list or `None`) with a console warning.
+- **`"ignore"`**: Returns an empty value silently.
+- **`"tolerate"`**: Fetches the data but logs a warning about potential performance issues.
+- **`"allow"`**: Fetches the data silently.
+
+If `lazy=None` (the default), the relationship uses the database's default lazy policy. You can set this globally via
+`TypeDAL`'s `lazy_policy` option (see [7. Advanced Configuration](./7_configuration.md) for configuration details),
+which defaults to `"tolerate"`.
+
+### Explicit Relationships
+
+Use `explicit=True` for relationships that are expensive to join or rarely needed:
+
+```python
+@db.define()
+class User(TypedTable):
+    name: str
+    audit_logs = relationship(list["AuditLog"], condition=lambda user, log: user.id == log.user, explicit=True)
+```
+
+When you call `.join()` without arguments, explicit relationships are skipped:
+
+```python
+user = User.join().first()  # user.audit_logs follows the lazy policy (empty/error/warning depending on setting)
+```
+
+To include an explicit relationship, reference it by name:
+
+```python
+user = User.join("audit_logs").first()  # now user.audit_logs is populated
+```
+
+## What's Next?
+
+Depending on your setup:
+
+- **Using py4web or web2py?** → [5. py4web & web2py](./5_py4web.md)
+- **Ready to manage your database?** → [6. Migrations](./6_migrations.md)
+- **Dive deeper into functionality?** → [8.: Mixins](./8_mixins.md)
+